2023-09-30T00:02:46.972 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T00:02:46.973 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{rhel_8} tasks/module_selftest} --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690 --owner scheduled_yuriw@teuthology --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690/orig.config.yaml 2023-09-30T00:02:47.009 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:02:47.088 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{rhel_8} tasks/module_selftest} email: yweinste@redhat.com first_in_suite: false job_id: '7406690' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 2 size: 30 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: client: debug client: 20 debug mgrc: 20 debug ms: 1 global: mon election default strategy: 3 mds: debug mgrc: 20 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mgr ttl cache expire seconds: 5 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug mgrc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mgr.x - mon.a - mon.c - mds.a - mds.c - osd.0 - client.0 - - mgr.y - mgr.z - mon.b - mds.b - osd.1 - osd.2 - osd.3 - client.1 seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMqbQIXsPQL+Ar4JA2DrEuvSb1C56pvU9IeQTqdHEFydStZM3Jh68Ce792vRf9g1fPDGyeeDxExrc5ud827ofuI= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG/8gFow/QHseLGfYRci2qn39BiLQNOI1wirw6FMtOPWoqfPf2NxV7uScS9oAVi0/HZb9BM2iGwESeViMP3MP7c= tasks: - install: null - ceph: log-ignorelist: - overall HEALTH_ - \(MGR_DOWN\) - \(PG_ - replacing it with standby - No standby daemons available - Reduced data availability - Degraded data redundancy - objects misplaced - Synthetic exception in serve - influxdb python module not found - \(MGR_ZABBIX_ - foo bar - Failed to open Telegraf - evicting unresponsive client - 1 mgr modules have recently crashed \(RECENT_MGR_MODULE_CRASH\) wait-for-scrub: false - cephfs_test_runner: fail_on_skip: false modules: - tasks.mgr.test_module_selftest teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T00:02:47.089 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T00:02:47.090 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T00:02:47.090 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T00:02:47.091 INFO:teuthology.task.internal:Checking packages... 2023-09-30T00:02:47.114 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T00:02:47.114 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T00:02:47.114 INFO:teuthology.packaging:ref: None 2023-09-30T00:02:47.115 INFO:teuthology.packaging:tag: None 2023-09-30T00:02:47.115 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:02:47.115 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:02:47.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:02:47.340 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-30T00:02:47.340 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T00:02:47.346 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T00:02:47.346 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T00:02:47.371 INFO:teuthology.task.internal:Saving configuration 2023-09-30T00:02:47.384 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T00:02:47.390 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:02:47.415 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 23:57:50.701634', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMqbQIXsPQL+Ar4JA2DrEuvSb1C56pvU9IeQTqdHEFydStZM3Jh68Ce792vRf9g1fPDGyeeDxExrc5ud827ofuI='} 2023-09-30T00:02:47.438 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 23:57:50.703510', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG/8gFow/QHseLGfYRci2qn39BiLQNOI1wirw6FMtOPWoqfPf2NxV7uScS9oAVi0/HZb9BM2iGwESeViMP3MP7c='} 2023-09-30T00:02:47.439 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T00:02:47.445 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-09-30T00:02:47.445 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-09-30T00:02:47.445 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T00:02:47.533 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f60f342dee0>, signals=[15]) 2023-09-30T00:02:47.533 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T00:02:47.539 INFO:teuthology.task.internal:Opening connections... 2023-09-30T00:02:47.540 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2023-09-30T00:02:47.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:02:47.611 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2023-09-30T00:02:47.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:02:47.679 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T00:02:47.686 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2023-09-30T00:02:47.703 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2023-09-30T00:02:47.703 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2023-09-30T00:02:47.759 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="Red Hat Enterprise Linux" 2023-09-30T00:02:47.759 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="8.4 (Ootpa)" 2023-09-30T00:02:47.760 INFO:teuthology.orchestra.run.smithi094.stdout:ID="rhel" 2023-09-30T00:02:47.760 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="fedora" 2023-09-30T00:02:47.760 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="8.4" 2023-09-30T00:02:47.760 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:02:47.760 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-09-30T00:02:47.760 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2023-09-30T00:02:47.760 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-09-30T00:02:47.760 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://www.redhat.com/" 2023-09-30T00:02:47.760 INFO:teuthology.orchestra.run.smithi094.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-30T00:02:47.760 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:02:47.760 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:02:47.761 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:02:47.761 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-09-30T00:02:47.761 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-30T00:02:47.761 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-09-30T00:02:47.761 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2023-09-30T00:02:47.782 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2023-09-30T00:02:47.799 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2023-09-30T00:02:47.799 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2023-09-30T00:02:47.855 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Red Hat Enterprise Linux" 2023-09-30T00:02:47.855 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8.4 (Ootpa)" 2023-09-30T00:02:47.855 INFO:teuthology.orchestra.run.smithi121.stdout:ID="rhel" 2023-09-30T00:02:47.855 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="fedora" 2023-09-30T00:02:47.855 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8.4" 2023-09-30T00:02:47.855 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:02:47.855 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-09-30T00:02:47.855 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2023-09-30T00:02:47.855 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-09-30T00:02:47.856 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.redhat.com/" 2023-09-30T00:02:47.856 INFO:teuthology.orchestra.run.smithi121.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-30T00:02:47.856 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:02:47.856 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:02:47.856 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:02:47.856 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-09-30T00:02:47.856 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-30T00:02:47.856 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-09-30T00:02:47.856 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2023-09-30T00:02:47.882 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T00:02:47.891 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T00:02:47.896 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T00:02:47.896 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:02:47.899 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:02:47.914 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T00:02:47.921 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T00:02:47.921 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:02:47.954 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:02:47.968 INFO:teuthology.orchestra.run.smithi094.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-09-30T00:02:47.971 INFO:teuthology.orchestra.run.smithi121.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-09-30T00:02:47.972 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T00:02:48.024 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T00:02:48.039 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T00:02:48.039 INFO:teuthology.task.kernel:config {'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T00:02:48.040 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:02:48.040 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:02:48.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:02:48.056 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2023-09-30T00:02:48.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:02:48.057 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2023-09-30T00:02:48.113 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-09-30T00:02:48.113 INFO:teuthology.task.kernel:Running kernel on smithi121: 4.18.0-305.25.1.el8_4.x86_64 2023-09-30T00:02:48.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2023-09-30T00:02:48.114 INFO:teuthology.orchestra.run.smithi094.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-09-30T00:02:48.115 INFO:teuthology.task.kernel:Running kernel on smithi094: 4.18.0-305.25.1.el8_4.x86_64 2023-09-30T00:02:48.115 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2023-09-30T00:02:49.559 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:02:49.560 INFO:teuthology.orchestra.run.smithi094.stdout:Unable to read consumer identity 2023-09-30T00:02:49.589 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:02:49.589 INFO:teuthology.orchestra.run.smithi121.stdout:Unable to read consumer identity 2023-09-30T00:02:50.948 INFO:teuthology.orchestra.run.smithi094.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-09-30T00:02:50.952 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2023-09-30T00:02:52.888 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 8.3 MB/s | 16 MB 00:01 2023-09-30T00:02:52.922 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 16 MB 00:01 2023-09-30T00:02:56.201 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 4.0 MB/s | 24 kB 00:00 2023-09-30T00:02:56.233 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-09-30T00:02:56.930 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-09-30T00:02:56.930 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-09-30T00:02:56.931 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-09-30T00:02:56.958 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-09-30T00:02:56.958 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-09-30T00:02:56.958 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-09-30T00:02:57.078 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:02:57.079 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-30T00:02:57.079 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:02:57.104 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:02:57.104 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2023-09-30T00:02:57.105 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:02:57.184 DEBUG:teuthology.orchestra.run.smithi121:> echo no | sudo yum reinstall kernel || true 2023-09-30T00:02:57.234 DEBUG:teuthology.orchestra.run.smithi094:> echo no | sudo yum reinstall kernel || true 2023-09-30T00:02:57.528 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:02:57.529 INFO:teuthology.orchestra.run.smithi121.stdout:Unable to read consumer identity 2023-09-30T00:02:57.576 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:02:57.577 INFO:teuthology.orchestra.run.smithi094.stdout:Unable to read consumer identity 2023-09-30T00:02:57.815 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:01 ago on Sat 30 Sep 2023 12:02:56 AM UTC. 2023-09-30T00:02:57.861 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:01 ago on Sat 30 Sep 2023 12:02:56 AM UTC. 2023-09-30T00:02:57.941 INFO:teuthology.orchestra.run.smithi121.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T00:02:57.941 INFO:teuthology.orchestra.run.smithi121.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:02:57.941 INFO:teuthology.orchestra.run.smithi121.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:02:57.969 INFO:teuthology.orchestra.run.smithi121.stderr:Error: No packages marked for reinstall. 2023-09-30T00:02:58.000 INFO:teuthology.orchestra.run.smithi094.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T00:02:58.000 INFO:teuthology.orchestra.run.smithi094.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:02:58.000 INFO:teuthology.orchestra.run.smithi094.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:02:58.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum reinstall -y kernel || true 2023-09-30T00:02:58.055 INFO:teuthology.orchestra.run.smithi094.stderr:Error: No packages marked for reinstall. 2023-09-30T00:02:58.104 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum reinstall -y kernel || true 2023-09-30T00:02:58.365 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:02:58.365 INFO:teuthology.orchestra.run.smithi121.stdout:Unable to read consumer identity 2023-09-30T00:02:58.446 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:02:58.446 INFO:teuthology.orchestra.run.smithi094.stdout:Unable to read consumer identity 2023-09-30T00:02:58.641 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:02 ago on Sat 30 Sep 2023 12:02:56 AM UTC. 2023-09-30T00:02:58.732 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:02 ago on Sat 30 Sep 2023 12:02:56 AM UTC. 2023-09-30T00:02:58.781 INFO:teuthology.orchestra.run.smithi121.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T00:02:58.781 INFO:teuthology.orchestra.run.smithi121.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:02:58.782 INFO:teuthology.orchestra.run.smithi121.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:02:58.795 INFO:teuthology.orchestra.run.smithi121.stderr:Error: No packages marked for reinstall. 2023-09-30T00:02:58.844 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:02:58.874 INFO:teuthology.orchestra.run.smithi094.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T00:02:58.874 INFO:teuthology.orchestra.run.smithi094.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:02:58.874 INFO:teuthology.orchestra.run.smithi094.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T00:02:58.896 INFO:teuthology.orchestra.run.smithi094.stderr:Error: No packages marked for reinstall. 2023-09-30T00:02:58.946 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:02:58.969 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:02:58.969 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:02:58.969 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:02:58.969 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:02:58.970 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:02:58.970 INFO:teuthology.task.kernel:Installing distro kernel on mgr.y... 2023-09-30T00:02:58.970 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:02:58.970 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:02:59.071 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:02:59.071 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:02:59.072 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:02:59.072 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:02:59.072 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:02:59.072 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2023-09-30T00:02:59.072 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:02:59.072 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:02:59.094 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:02:59.094 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:02:59.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2023-09-30T00:02:59.121 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2023-09-30T00:02:59.121 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2023-09-30T00:02:59.121 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.02 2023-09-30T00:02:59.121 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 99.el8_4.1 2023-09-30T00:02:59.122 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2023-09-30T00:02:59.122 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-09-30T00:02:59.122 INFO:teuthology.orchestra.run.smithi121.stdout:Group : System Environment/Base 2023-09-30T00:02:59.122 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 9264631 2023-09-30T00:02:59.122 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2023-09-30T00:02:59.122 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-09-30T00:02:59.122 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-09-30T00:02:59.122 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-09-30T00:02:59.122 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-09-30T00:02:59.122 INFO:teuthology.orchestra.run.smithi121.stdout:Relocations : (not relocatable) 2023-09-30T00:02:59.122 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : Red Hat, Inc. 2023-09-30T00:02:59.123 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : Red Hat, Inc. 2023-09-30T00:02:59.123 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:02:59.123 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2023-09-30T00:02:59.123 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2023-09-30T00:02:59.123 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:02:59.123 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:02:59.123 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:02:59.123 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:02:59.123 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2023-09-30T00:02:59.123 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:02:59.123 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:02:59.124 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:02:59.124 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:02:59.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:02:59.198 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:02:59.199 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:02:59.199 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -qi grub2-tools 2023-09-30T00:02:59.226 INFO:teuthology.orchestra.run.smithi094.stdout:Name : grub2-tools 2023-09-30T00:02:59.226 INFO:teuthology.orchestra.run.smithi094.stdout:Epoch : 1 2023-09-30T00:02:59.226 INFO:teuthology.orchestra.run.smithi094.stdout:Version : 2.02 2023-09-30T00:02:59.226 INFO:teuthology.orchestra.run.smithi094.stdout:Release : 99.el8_4.1 2023-09-30T00:02:59.226 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: x86_64 2023-09-30T00:02:59.226 INFO:teuthology.orchestra.run.smithi094.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-09-30T00:02:59.226 INFO:teuthology.orchestra.run.smithi094.stdout:Group : System Environment/Base 2023-09-30T00:02:59.226 INFO:teuthology.orchestra.run.smithi094.stdout:Size : 9264631 2023-09-30T00:02:59.227 INFO:teuthology.orchestra.run.smithi094.stdout:License : GPLv3+ 2023-09-30T00:02:59.227 INFO:teuthology.orchestra.run.smithi094.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-09-30T00:02:59.227 INFO:teuthology.orchestra.run.smithi094.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-09-30T00:02:59.227 INFO:teuthology.orchestra.run.smithi094.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-09-30T00:02:59.227 INFO:teuthology.orchestra.run.smithi094.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-09-30T00:02:59.227 INFO:teuthology.orchestra.run.smithi094.stdout:Relocations : (not relocatable) 2023-09-30T00:02:59.227 INFO:teuthology.orchestra.run.smithi094.stdout:Packager : Red Hat, Inc. 2023-09-30T00:02:59.227 INFO:teuthology.orchestra.run.smithi094.stdout:Vendor : Red Hat, Inc. 2023-09-30T00:02:59.227 INFO:teuthology.orchestra.run.smithi094.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:02:59.227 INFO:teuthology.orchestra.run.smithi094.stdout:Summary : Support tools for GRUB. 2023-09-30T00:02:59.227 INFO:teuthology.orchestra.run.smithi094.stdout:Description : 2023-09-30T00:02:59.228 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:02:59.228 INFO:teuthology.orchestra.run.smithi094.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:02:59.228 INFO:teuthology.orchestra.run.smithi094.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:02:59.228 INFO:teuthology.orchestra.run.smithi094.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:02:59.228 INFO:teuthology.orchestra.run.smithi094.stdout:hardware devices. 2023-09-30T00:02:59.228 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:02:59.228 INFO:teuthology.orchestra.run.smithi094.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:02:59.229 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:02:59.229 INFO:teuthology.task.kernel:Updating grub on smithi094 to boot 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:02:59.229 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:03:00.550 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2023-09-30T00:03:00.652 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2023-09-30T00:03:02.438 INFO:teuthology.orchestra.run.smithi121.stderr:done 2023-09-30T00:03:02.439 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2023-09-30T00:03:02.460 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.yewaJHDlK6 2023-09-30T00:03:02.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yewaJHDlK6 2023-09-30T00:03:02.482 INFO:teuthology.orchestra.run.smithi094.stderr:done 2023-09-30T00:03:02.483 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2023-09-30T00:03:02.512 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.uIf6JV0L9C 2023-09-30T00:03:02.512 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uIf6JV0L9C 2023-09-30T00:03:02.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.yewaJHDlK6 2023-09-30T00:03:02.582 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.uIf6JV0L9C 2023-09-30T00:03:02.696 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.yewaJHDlK6 is 6KB 2023-09-30T00:03:02.744 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.yewaJHDlK6 2023-09-30T00:03:02.753 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.uIf6JV0L9C is 6KB 2023-09-30T00:03:02.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:03:02.802 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.uIf6JV0L9C 2023-09-30T00:03:02.815 DEBUG:teuthology.orchestra.run.smithi094:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:03:02.828 INFO:teuthology.orchestra.run.smithi121.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-09-30T00:03:02.828 INFO:teuthology.orchestra.run.smithi121.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-09-30T00:03:02.828 INFO:teuthology.orchestra.run.smithi121.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-09-30T00:03:02.828 INFO:teuthology.orchestra.run.smithi121.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-09-30T00:03:02.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:03:02.887 INFO:teuthology.orchestra.run.smithi094.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-09-30T00:03:02.887 INFO:teuthology.orchestra.run.smithi094.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-09-30T00:03:02.887 INFO:teuthology.orchestra.run.smithi094.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-09-30T00:03:02.887 INFO:teuthology.orchestra.run.smithi094.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-09-30T00:03:02.888 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-09-30T00:03:02.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2023-09-30T00:03:03.071 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2023-09-30T00:03:33.022 INFO:teuthology.task.kernel:Checking client mgr.y for new kernel version... 2023-09-30T00:03:33.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-09-30T00:03:33.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:03:33.074 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2023-09-30T00:03:33.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2023-09-30T00:03:33.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:04:33.083 DEBUG:teuthology.orchestra.remote:timed out 2023-09-30T00:04:33.084 DEBUG:teuthology.orchestra.remote:timed out 2023-09-30T00:04:42.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-09-30T00:04:42.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:04:42.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2023-09-30T00:04:42.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:04:42.562 DEBUG:teuthology.orchestra.run.smithi094:> true 2023-09-30T00:04:42.885 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2023-09-30T00:04:42.885 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "4.18.0-425.3.1.el8.x86_64"... 2023-09-30T00:04:42.886 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2023-09-30T00:04:42.900 INFO:teuthology.orchestra.run.smithi094.stdout:4.18.0-425.3.1.el8.x86_64 2023-09-30T00:04:42.900 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:04:42.901 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:04:42.901 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-09-30T00:04:43.902 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2023-09-30T00:04:43.902 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:04:44.146 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2023-09-30T00:04:44.166 DEBUG:teuthology.parallel:result is None 2023-09-30T00:04:45.504 DEBUG:teuthology.orchestra.run.smithi121:> true 2023-09-30T00:04:45.914 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-09-30T00:04:45.914 INFO:teuthology.task.kernel:Checking kernel version of mgr.y, want "4.18.0-425.3.1.el8.x86_64"... 2023-09-30T00:04:45.914 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2023-09-30T00:04:45.928 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-425.3.1.el8.x86_64 2023-09-30T00:04:45.928 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-09-30T00:04:45.928 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:04:45.928 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-09-30T00:04:46.929 INFO:teuthology.task.kernel:Enabling kdb on mgr.y... 2023-09-30T00:04:46.929 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:04:47.086 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2023-09-30T00:04:47.104 DEBUG:teuthology.parallel:result is None 2023-09-30T00:04:47.104 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T00:04:47.112 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T00:04:47.112 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:04:47.115 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:04:47.129 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T00:04:47.135 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T00:04:47.143 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T00:04:47.144 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:04:47.170 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:04:47.204 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T00:04:47.211 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T00:04:47.211 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:04:47.242 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:04:47.270 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:04:47.276 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:04:47.281 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:04:47.287 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:04:47.289 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T00:04:47.295 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T00:04:47.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:04:47.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:04:47.354 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T00:04:47.364 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T00:04:47.365 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:04:47.392 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:04:47.408 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:04:47.488 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:04:47.546 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:04:47.626 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:04:47.672 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:04:47.672 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:04:47.736 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:04:47.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:04:47.823 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:04:47.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:04:47.946 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:04:47.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:04:48.022 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2023-09-30T00:04:48.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2023-09-30T00:04:48.123 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:04:48.194 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:04:48.244 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T00:04:48.257 INFO:teuthology.task.internal:Starting timer... 2023-09-30T00:04:48.257 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T00:04:48.275 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T00:04:48.283 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-30T00:04:48.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2023-09-30T00:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: [ OK ] 2023-09-30T00:04:48.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2023-09-30T00:04:48.664 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: [ OK ] 2023-09-30T00:04:48.666 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T00:04:48.666 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2023-09-30T00:04:48.748 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2023-09-30T00:04:48.749 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2023-09-30T00:04:48.773 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2023-09-30T00:04:48.773 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi094.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive'} 2023-09-30T00:04:48.773 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:04:49.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:04:49.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:04:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:04:49.191 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T00:04:49.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:04:49.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:04:49.669 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T00:04:49.686 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-30T00:04:49.686 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T00:04:49.703 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T00:04:49.704 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi094.front.sepia.ceph.com,smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T00:12:53.497 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi094.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2023-09-30T00:12:53.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2023-09-30T00:12:53.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:12:53.569 DEBUG:teuthology.orchestra.run.smithi094:> true 2023-09-30T00:12:53.591 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2023-09-30T00:12:53.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-09-30T00:12:53.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:12:53.662 DEBUG:teuthology.orchestra.run.smithi121:> true 2023-09-30T00:12:53.725 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-09-30T00:12:53.726 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T00:12:53.737 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T00:12:53.737 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:12:53.738 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:12:53.740 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:12:53.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:12:53.766 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T00:12:53.780 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T00:12:53.795 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T00:12:53.809 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T00:12:53.817 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2023-09-30T00:12:53.827 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2023-09-30T00:12:53.841 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T00:12:53.842 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2023-09-30T00:12:53.852 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2023-09-30T00:12:53.854 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T00:12:53.865 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T00:12:53.879 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T00:12:53.926 INFO:teuthology.orchestra.run.smithi094.stderr:bash: ntpq: command not found 2023-09-30T00:12:53.929 INFO:teuthology.orchestra.run.smithi094.stdout:210 Number of sources = 4 2023-09-30T00:12:53.929 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:12:53.929 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2023-09-30T00:12:53.929 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:12:53.929 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:12:53.930 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:12:53.930 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:12:53.940 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2023-09-30T00:12:53.943 INFO:teuthology.orchestra.run.smithi121.stdout:210 Number of sources = 4 2023-09-30T00:12:53.943 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:12:53.943 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2023-09-30T00:12:53.943 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:12:53.943 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:12:53.943 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:12:53.943 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:12:53.945 INFO:teuthology.run_tasks:Running task install... 2023-09-30T00:12:53.955 DEBUG:teuthology.task.install:project ceph 2023-09-30T00:12:53.955 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T00:12:53.955 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T00:12:53.955 INFO:teuthology.task.install:Using flavor: default 2023-09-30T00:12:53.962 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T00:12:53.962 INFO:teuthology.task.install:extra packages: [] 2023-09-30T00:12:53.962 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T00:12:53.963 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:12:53.965 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T00:12:53.965 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:12:54.196 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T00:12:54.196 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T00:12:54.201 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T00:12:54.202 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T00:12:54.309 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T00:12:54.310 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:12:54.310 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T00:12:54.336 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T00:12:54.336 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T00:12:54.360 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T00:12:54.361 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:12:54.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T00:12:54.386 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T00:12:54.386 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T00:12:54.401 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T00:12:54.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T00:12:54.475 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T00:12:54.526 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T00:12:54.539 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2023-09-30T00:12:54.540 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2023-09-30T00:12:54.589 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2023-09-30T00:12:54.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2023-09-30T00:12:54.929 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:12:54.981 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:12:55.286 INFO:teuthology.orchestra.run.smithi094.stdout:44 files removed 2023-09-30T00:12:55.303 INFO:teuthology.orchestra.run.smithi121.stdout:44 files removed 2023-09-30T00:12:55.341 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw 2023-09-30T00:12:55.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2023-09-30T00:12:55.689 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:12:55.700 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:12:56.493 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 211 kB/s | 78 kB 00:00 2023-09-30T00:12:56.502 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 211 kB/s | 78 kB 00:00 2023-09-30T00:12:56.843 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 30 kB/s | 9.2 kB 00:00 2023-09-30T00:12:56.845 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 30 kB/s | 9.2 kB 00:00 2023-09-30T00:12:57.116 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 6.1 kB/s | 1.5 kB 00:00 2023-09-30T00:12:57.117 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 6.0 kB/s | 1.5 kB 00:00 2023-09-30T00:12:57.225 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 44 kB/s | 3.5 kB 00:00 2023-09-30T00:12:57.225 INFO:teuthology.orchestra.run.smithi094.stdout:Copr repo for python3-asyncssh owned by ceph 45 kB/s | 3.5 kB 00:00 2023-09-30T00:12:57.968 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 22 MB/s | 16 MB 00:00 2023-09-30T00:13:00.878 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 16 MB 00:03 2023-09-30T00:13:01.317 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 2.8 MB/s | 24 kB 00:00 2023-09-30T00:13:01.668 INFO:teuthology.orchestra.run.smithi121.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 122 MB/s | 33 MB 00:00 2023-09-30T00:13:04.135 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2023-09-30T00:13:04.489 INFO:teuthology.orchestra.run.smithi094.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 120 MB/s | 33 MB 00:00 2023-09-30T00:13:06.802 INFO:teuthology.orchestra.run.smithi121.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 129 MB/s | 38 MB 00:00 2023-09-30T00:13:09.595 INFO:teuthology.orchestra.run.smithi094.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 127 MB/s | 38 MB 00:00 2023-09-30T00:13:15.858 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:13:15.861 INFO:teuthology.orchestra.run.smithi121.stdout:==================================================================================================== 2023-09-30T00:13:15.861 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-30T00:13:15.861 INFO:teuthology.orchestra.run.smithi121.stdout:==================================================================================================== 2023-09-30T00:13:15.861 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:13:15.862 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:13:15.862 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2023-09-30T00:13:15.862 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:13:15.862 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T00:13:15.862 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-30T00:13:15.862 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T00:13:15.862 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T00:13:15.862 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T00:13:15.862 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T00:13:15.862 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T00:13:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T00:13:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-30T00:13:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T00:13:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-09-30T00:13:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T00:13:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T00:13:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-30T00:13:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T00:13:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T00:13:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T00:13:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-30T00:13:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T00:13:15.864 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T00:13:15.864 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T00:13:15.864 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T00:13:15.864 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:13:15.864 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:13:15.864 INFO:teuthology.orchestra.run.smithi121.stdout:==================================================================================================== 2023-09-30T00:13:15.864 INFO:teuthology.orchestra.run.smithi121.stdout:Install 21 Packages 2023-09-30T00:13:15.864 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2023-09-30T00:13:15.864 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:13:15.865 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 63 M 2023-09-30T00:13:15.865 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:13:16.908 INFO:teuthology.orchestra.run.smithi121.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 7.2 MB/s | 5.2 MB 00:00 2023-09-30T00:13:16.950 INFO:teuthology.orchestra.run.smithi121.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 578 kB/s | 24 kB 00:00 2023-09-30T00:13:17.020 INFO:teuthology.orchestra.run.smithi121.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 11 MB/s | 796 kB 00:00 2023-09-30T00:13:17.079 INFO:teuthology.orchestra.run.smithi121.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 8.4 MB/s | 506 kB 00:00 2023-09-30T00:13:17.537 INFO:teuthology.orchestra.run.smithi121.stdout:(5/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 12 MB/s | 5.6 MB 00:00 2023-09-30T00:13:17.584 INFO:teuthology.orchestra.run.smithi121.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 957 kB/s | 45 kB 00:00 2023-09-30T00:13:17.633 INFO:teuthology.orchestra.run.smithi121.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 2.3 MB/s | 115 kB 00:00 2023-09-30T00:13:17.684 INFO:teuthology.orchestra.run.smithi121.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 3.6 MB/s | 190 kB 00:00 2023-09-30T00:13:17.742 INFO:teuthology.orchestra.run.smithi121.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 6.3 MB/s | 370 kB 00:00 2023-09-30T00:13:17.800 INFO:teuthology.orchestra.run.smithi121.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 6.0 MB/s | 358 kB 00:00 2023-09-30T00:13:17.849 INFO:teuthology.orchestra.run.smithi121.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 2.1 MB/s | 107 kB 00:00 2023-09-30T00:13:18.167 INFO:teuthology.orchestra.run.smithi121.stdout:(12/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8 8.5 MB/s | 17 MB 00:01 2023-09-30T00:13:18.169 INFO:teuthology.orchestra.run.smithi121.stdout:(13/23): gperftools-libs-2.7-9.el8.x86_64.rpm 956 kB/s | 306 kB 00:00 2023-09-30T00:13:18.206 INFO:teuthology.orchestra.run.smithi121.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.0 MB/s | 75 kB 00:00 2023-09-30T00:13:18.307 INFO:teuthology.orchestra.run.smithi121.stdout:(15/23): liboath-2.6.2-3.el8.x86_64.rpm 421 kB/s | 59 kB 00:00 2023-09-30T00:13:18.329 INFO:teuthology.orchestra.run.smithi121.stdout:(16/23): thrift-0.13.0-2.el8.x86_64.rpm 14 MB/s | 1.7 MB 00:00 2023-09-30T00:13:18.343 INFO:teuthology.orchestra.run.smithi121.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 1.2 MB/s | 44 kB 00:00 2023-09-30T00:13:18.363 INFO:teuthology.orchestra.run.smithi121.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.4 MB/s | 259 kB 00:00 2023-09-30T00:13:18.366 INFO:teuthology.orchestra.run.smithi121.stdout:(19/23): librdkafka-0.11.4-1.el8.x86_64.rpm 15 MB/s | 353 kB 00:00 2023-09-30T00:13:18.374 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:13:18.377 INFO:teuthology.orchestra.run.smithi094.stdout:==================================================================================================== 2023-09-30T00:13:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2023-09-30T00:13:18.377 INFO:teuthology.orchestra.run.smithi094.stdout:==================================================================================================== 2023-09-30T00:13:18.377 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:13:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:13:18.377 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2023-09-30T00:13:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:13:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T00:13:18.377 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2023-09-30T00:13:18.377 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T00:13:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T00:13:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T00:13:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T00:13:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T00:13:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T00:13:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-30T00:13:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T00:13:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-09-30T00:13:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T00:13:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T00:13:18.378 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-30T00:13:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T00:13:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T00:13:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T00:13:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-30T00:13:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T00:13:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T00:13:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T00:13:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T00:13:18.379 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:13:18.379 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:13:18.379 INFO:teuthology.orchestra.run.smithi094.stdout:==================================================================================================== 2023-09-30T00:13:18.379 INFO:teuthology.orchestra.run.smithi094.stdout:Install 21 Packages 2023-09-30T00:13:18.380 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 2 Packages 2023-09-30T00:13:18.380 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:13:18.380 INFO:teuthology.orchestra.run.smithi121.stdout:(20/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.2 MB/s | 47 kB 00:00 2023-09-30T00:13:18.380 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 63 M 2023-09-30T00:13:18.381 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:13:18.709 INFO:teuthology.orchestra.run.smithi121.stdout:(21/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 9.3 MB/s | 23 MB 00:02 2023-09-30T00:13:18.788 INFO:teuthology.orchestra.run.smithi121.stdout:(22/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 8.2 MB/s | 3.5 MB 00:00 2023-09-30T00:13:18.800 INFO:teuthology.orchestra.run.smithi121.stdout:(23/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 8.5 MB/s | 3.6 MB 00:00 2023-09-30T00:13:18.803 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:13:18.803 INFO:teuthology.orchestra.run.smithi121.stdout:Total 22 MB/s | 63 MB 00:02 2023-09-30T00:13:18.807 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:13:18.850 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:13:18.850 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:13:19.619 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:13:19.623 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:13:20.287 INFO:teuthology.orchestra.run.smithi094.stdout:(1/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 9.6 MB/s | 17 MB 00:01 2023-09-30T00:13:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 517 kB/s | 24 kB 00:00 2023-09-30T00:13:20.409 INFO:teuthology.orchestra.run.smithi094.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 11 MB/s | 796 kB 00:00 2023-09-30T00:13:20.576 INFO:teuthology.orchestra.run.smithi094.stdout:(4/23): ceph-common-17.2.6-1165.g93dce5e1.el8.x 11 MB/s | 23 MB 00:02 2023-09-30T00:13:20.580 INFO:teuthology.orchestra.run.smithi094.stdout:(5/23): libradosstriper1-17.2.6-1165.g93dce5e1. 2.9 MB/s | 506 kB 00:00 2023-09-30T00:13:20.645 INFO:teuthology.orchestra.run.smithi094.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 697 kB/s | 45 kB 00:00 2023-09-30T00:13:20.695 INFO:teuthology.orchestra.run.smithi094.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 2.3 MB/s | 115 kB 00:00 2023-09-30T00:13:20.748 INFO:teuthology.orchestra.run.smithi094.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 3.6 MB/s | 190 kB 00:00 2023-09-30T00:13:20.797 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:13:20.805 INFO:teuthology.orchestra.run.smithi094.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 6.3 MB/s | 370 kB 00:00 2023-09-30T00:13:20.852 INFO:teuthology.orchestra.run.smithi094.stdout:(10/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_ 20 MB/s | 5.6 MB 00:00 2023-09-30T00:13:20.854 INFO:teuthology.orchestra.run.smithi094.stdout:(11/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 7.1 MB/s | 358 kB 00:00 2023-09-30T00:13:20.889 INFO:teuthology.orchestra.run.smithi094.stdout:(12/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 2.8 MB/s | 107 kB 00:00 2023-09-30T00:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T00:13:21.058 INFO:teuthology.orchestra.run.smithi094.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 348 kB/s | 59 kB 00:00 2023-09-30T00:13:21.107 INFO:teuthology.orchestra.run.smithi094.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2023-09-30T00:13:21.120 INFO:teuthology.orchestra.run.smithi094.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.2 MB/s | 75 kB 00:00 2023-09-30T00:13:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 1.4 MB/s | 44 kB 00:00 2023-09-30T00:13:21.172 INFO:teuthology.orchestra.run.smithi094.stdout:(17/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x8 2.0 MB/s | 5.2 MB 00:02 2023-09-30T00:13:21.176 INFO:teuthology.orchestra.run.smithi094.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 10 MB/s | 259 kB 00:00 2023-09-30T00:13:21.193 INFO:teuthology.orchestra.run.smithi094.stdout:(19/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.9 MB/s | 47 kB 00:00 2023-09-30T00:13:21.205 INFO:teuthology.orchestra.run.smithi094.stdout:(20/23): librdkafka-0.11.4-1.el8.x86_64.rpm 11 MB/s | 353 kB 00:00 2023-09-30T00:13:21.277 INFO:teuthology.orchestra.run.smithi094.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 10 MB/s | 1.7 MB 00:00 2023-09-30T00:13:21.390 INFO:teuthology.orchestra.run.smithi094.stdout:(22/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 19 MB/s | 3.6 MB 00:00 2023-09-30T00:13:21.416 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T00:13:21.644 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:13:21.751 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:13:21.888 INFO:teuthology.orchestra.run.smithi094.stdout:(23/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 5.0 MB/s | 3.5 MB 00:00 2023-09-30T00:13:21.894 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:13:21.894 INFO:teuthology.orchestra.run.smithi094.stdout:Total 18 MB/s | 63 MB 00:03 2023-09-30T00:13:21.898 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:13:21.933 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:13:21.942 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:13:21.943 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:13:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-09-30T00:13:22.452 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:13:22.663 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:13:22.668 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:13:22.839 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:13:23.035 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:13:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:13:23.353 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2023-09-30T00:13:23.577 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2023-09-30T00:13:23.832 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2023-09-30T00:13:23.888 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:13:24.105 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T00:13:24.493 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T00:13:24.518 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-09-30T00:13:24.738 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:13:24.751 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:13:24.822 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:13:24.848 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:13:24.937 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:13:25.036 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:13:25.154 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:13:25.313 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-09-30T00:13:25.395 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:13:25.546 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:13:25.630 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:13:25.768 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:13:25.852 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:13:25.937 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:13:26.028 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T00:13:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:13:26.239 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:13:26.270 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-09-30T00:13:26.462 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2023-09-30T00:13:26.533 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T00:13:26.705 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2023-09-30T00:13:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:13:26.907 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2023-09-30T00:13:27.600 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-09-30T00:13:27.796 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:13:27.861 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:13:27.998 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:13:28.224 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:13:28.507 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:13:28.766 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:13:28.955 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:13:29.024 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:13:29.222 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T00:13:29.405 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-09-30T00:13:29.683 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T00:13:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:13:31.776 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:13:32.017 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:13:32.642 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:13:32.967 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:13:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:13:33.208 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:13:34.892 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:13:35.145 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:13:35.766 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:13:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:13:36.228 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:13:36.387 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:13:51.123 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:13:51.124 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2023-09-30T00:13:51.124 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2023-09-30T00:13:51.124 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2023-09-30T00:13:51.124 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /run 2023-09-30T00:13:51.124 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2023-09-30T00:13:51.124 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2023-09-30T00:13:51.124 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2023-09-30T00:13:51.124 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2023-09-30T00:13:51.124 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2023-09-30T00:13:51.124 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:13:52.863 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:13:52.925 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:13:52.925 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:13:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:13:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:13:53.936 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:13:53.936 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T00:13:53.936 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:13:53.936 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:13:53.936 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:13:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:13:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T00:13:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T00:13:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T00:13:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T00:13:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:13:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T00:13:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:13:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:13:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-09-30T00:13:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-09-30T00:13:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:13:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-09-30T00:13:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T00:13:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-09-30T00:13:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 20/25 2023-09-30T00:13:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2023-09-30T00:13:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:13:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T00:13:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T00:13:54.335 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:13:54.335 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2023-09-30T00:13:54.335 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2023-09-30T00:13:54.335 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /dev 2023-09-30T00:13:54.335 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /run 2023-09-30T00:13:54.335 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2023-09-30T00:13:54.335 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2023-09-30T00:13:54.335 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2023-09-30T00:13:54.336 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2023-09-30T00:13:54.336 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /dev 2023-09-30T00:13:54.336 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:13:55.462 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:13:55.463 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:49 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:13:56.083 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:13:56.146 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:13:56.147 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:13:56.378 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:13:56.378 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:13:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:13:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T00:13:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:13:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:13:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:13:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:13:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T00:13:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T00:13:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T00:13:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T00:13:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:13:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T00:13:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:13:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:13:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-09-30T00:13:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-09-30T00:13:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:13:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-09-30T00:13:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T00:13:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-09-30T00:13:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 20/25 2023-09-30T00:13:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2023-09-30T00:13:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:13:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T00:13:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T00:13:57.334 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:13:57.444 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:13:57.445 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2023-09-30T00:13:57.445 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.445 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.445 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:13:57.445 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.445 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.445 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.446 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.446 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T00:13:57.446 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.446 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T00:13:57.446 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-30T00:13:57.446 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.446 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-09-30T00:13:57.446 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.446 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T00:13:57.446 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T00:13:57.446 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T00:13:57.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.447 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:13:57.447 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T00:13:57.447 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:13:57.447 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:13:57.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2023-09-30T00:13:57.961 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:13:58.743 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:13:58.744 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:49 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:13:58.796 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:52 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:14:00.641 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:14:00.641 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2023-09-30T00:14:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-30T00:14:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2023-09-30T00:14:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:14:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T00:14:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-30T00:14:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-09-30T00:14:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T00:14:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-30T00:14:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-09-30T00:14:00.642 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-30T00:14:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:14:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2023-09-30T00:14:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2023-09-30T00:14:00.643 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 53 M 2023-09-30T00:14:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 233 M 2023-09-30T00:14:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:14:00.648 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 18 MB/s | 70 kB 00:00 2023-09-30T00:14:00.677 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 5.4 MB/s | 161 kB 00:00 2023-09-30T00:14:00.680 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:14:00.692 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 13 MB/s | 187 kB 00:00 2023-09-30T00:14:00.708 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 19 MB/s | 302 kB 00:00 2023-09-30T00:14:00.790 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:00.790 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2023-09-30T00:14:00.790 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.791 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:14:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T00:14:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T00:14:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-30T00:14:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-09-30T00:14:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T00:14:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T00:14:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T00:14:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.793 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:00.793 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T00:14:00.793 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:00.793 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:14:00.796 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 1.1 MB/s | 175 kB 00:00 2023-09-30T00:14:00.952 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-test 2023-09-30T00:14:01.315 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:14:02.179 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:53 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:14:03.197 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 20 MB/s | 52 MB 00:02 2023-09-30T00:14:03.199 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:14:03.200 INFO:teuthology.orchestra.run.smithi121.stdout:Total 21 MB/s | 53 MB 00:02 2023-09-30T00:14:03.204 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:14:03.227 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:14:03.228 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:14:03.598 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:14:03.602 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:14:04.021 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:14:04.023 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2023-09-30T00:14:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2023-09-30T00:14:04.023 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2023-09-30T00:14:04.023 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:14:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T00:14:04.023 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2023-09-30T00:14:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-09-30T00:14:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T00:14:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-30T00:14:04.023 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-09-30T00:14:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-30T00:14:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:04.024 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:14:04.024 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2023-09-30T00:14:04.024 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2023-09-30T00:14:04.024 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:04.024 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 53 M 2023-09-30T00:14:04.025 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 233 M 2023-09-30T00:14:04.025 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:14:04.029 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 18 MB/s | 70 kB 00:00 2023-09-30T00:14:04.062 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 4.9 MB/s | 161 kB 00:00 2023-09-30T00:14:04.083 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 11 MB/s | 187 kB 00:00 2023-09-30T00:14:04.101 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 16 MB/s | 302 kB 00:00 2023-09-30T00:14:04.139 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:14:04.306 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 622 kB/s | 175 kB 00:00 2023-09-30T00:14:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-09-30T00:14:04.618 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T00:14:04.699 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T00:14:04.901 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-09-30T00:14:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-09-30T00:14:05.243 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:14:06.376 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 22 MB/s | 52 MB 00:02 2023-09-30T00:14:06.380 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:14:06.380 INFO:teuthology.orchestra.run.smithi094.stdout:Total 22 MB/s | 53 MB 00:02 2023-09-30T00:14:06.384 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:14:06.408 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:14:06.408 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:14:06.817 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:14:06.821 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:14:07.358 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:14:07.604 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-09-30T00:14:07.841 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T00:14:07.912 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T00:14:08.114 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-09-30T00:14:08.316 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-09-30T00:14:08.491 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:14:10.267 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:14:10.557 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:14:10.812 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:14:10.812 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T00:14:10.812 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T00:14:10.813 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-09-30T00:14:10.813 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-09-30T00:14:10.813 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-09-30T00:14:11.852 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-09-30T00:14:11.852 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:05 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:14:13.490 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:14:13.726 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:14:13.823 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:14:13.824 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:13.824 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:14:13.824 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:13.824 INFO:teuthology.orchestra.run.smithi121.stdout: jq-1.5-12.el8.x86_64 2023-09-30T00:14:13.824 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:13.824 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T00:14:13.824 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.3.3-2.el8.x86_64 2023-09-30T00:14:13.824 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T00:14:13.825 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:13.825 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:14:13.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2023-09-30T00:14:14.094 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:14:14.095 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T00:14:14.095 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T00:14:14.095 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-09-30T00:14:14.095 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-09-30T00:14:14.095 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-09-30T00:14:14.316 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:14:15.234 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-09-30T00:14:15.234 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:06 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:14:15.247 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:09 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:14:17.090 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:14:17.093 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================================= 2023-09-30T00:14:17.094 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-30T00:14:17.094 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================================= 2023-09-30T00:14:17.094 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:14:17.094 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T00:14:17.094 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-30T00:14:17.094 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T00:14:17.094 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T00:14:17.095 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T00:14:17.095 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T00:14:17.095 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:14:17.095 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T00:14:17.095 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T00:14:17.095 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T00:14:17.095 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T00:14:17.095 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T00:14:17.095 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T00:14:17.095 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T00:14:17.095 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-30T00:14:17.096 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-30T00:14:17.096 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T00:14:17.096 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T00:14:17.096 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T00:14:17.096 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-30T00:14:17.096 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T00:14:17.096 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T00:14:17.096 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T00:14:17.096 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T00:14:17.096 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T00:14:17.096 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T00:14:17.097 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T00:14:17.097 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-30T00:14:17.097 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T00:14:17.097 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:17.097 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:14:17.097 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================================= 2023-09-30T00:14:17.097 INFO:teuthology.orchestra.run.smithi121.stdout:Install 28 Packages 2023-09-30T00:14:17.097 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:17.098 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 27 M 2023-09-30T00:14:17.098 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 106 M 2023-09-30T00:14:17.098 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:14:17.190 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:14:17.287 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:17.288 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:14:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: jq-1.5-12.el8.x86_64 2023-09-30T00:14:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T00:14:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.3.3-2.el8.x86_64 2023-09-30T00:14:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T00:14:17.288 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:17.288 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:14:17.327 INFO:teuthology.orchestra.run.smithi121.stdout:(1/28): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 42 kB/s | 6.4 kB 00:00 2023-09-30T00:14:17.409 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph 2023-09-30T00:14:17.414 INFO:teuthology.orchestra.run.smithi121.stdout:(2/28): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 6.6 MB/s | 1.6 MB 00:00 2023-09-30T00:14:17.764 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:14:17.954 INFO:teuthology.orchestra.run.smithi121.stdout:(3/28): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 3.0 MB/s | 2.3 MB 00:00 2023-09-30T00:14:18.030 INFO:teuthology.orchestra.run.smithi121.stdout:(4/28): ceph-mgr-modules-core-17.2.6-1165.g93dc 3.1 MB/s | 238 kB 00:00 2023-09-30T00:14:18.094 INFO:teuthology.orchestra.run.smithi121.stdout:(5/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 688 kB/s | 44 kB 00:00 2023-09-30T00:14:18.128 INFO:teuthology.orchestra.run.smithi121.stdout:(6/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 5.4 MB/s | 185 kB 00:00 2023-09-30T00:14:18.142 INFO:teuthology.orchestra.run.smithi121.stdout:(7/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 12 MB/s | 173 kB 00:00 2023-09-30T00:14:18.253 INFO:teuthology.orchestra.run.smithi121.stdout:(8/28): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86_ 20 MB/s | 17 MB 00:00 2023-09-30T00:14:18.257 INFO:teuthology.orchestra.run.smithi121.stdout:(9/28): python3-cherrypy-18.4.0-1.el8.noarch.rp 3.3 MB/s | 384 kB 00:00 2023-09-30T00:14:18.270 INFO:teuthology.orchestra.run.smithi121.stdout:(10/28): python3-jaraco-functools-2.0-4.el8.noa 1.3 MB/s | 18 kB 00:00 2023-09-30T00:14:18.282 INFO:teuthology.orchestra.run.smithi121.stdout:(11/28): python3-logutils-0.3.5-11.el8.noarch.r 4.0 MB/s | 49 kB 00:00 2023-09-30T00:14:18.295 INFO:teuthology.orchestra.run.smithi121.stdout:(12/28): python3-more-itertools-7.2.0-3.el8.noa 4.7 MB/s | 59 kB 00:00 2023-09-30T00:14:18.306 INFO:teuthology.orchestra.run.smithi121.stdout:(13/28): python3-jaraco-6.2-6.el8.noarch.rpm 213 kB/s | 11 kB 00:00 2023-09-30T00:14:18.311 INFO:teuthology.orchestra.run.smithi121.stdout:(14/28): python3-pecan-1.3.2-9.el8.noarch.rpm 18 MB/s | 283 kB 00:00 2023-09-30T00:14:18.317 INFO:teuthology.orchestra.run.smithi121.stdout:(15/28): python3-portend-2.6-1.el8.noarch.rpm 1.3 MB/s | 16 kB 00:00 2023-09-30T00:14:18.323 INFO:teuthology.orchestra.run.smithi121.stdout:(16/28): python3-simplegeneric-0.8.1-17.el8.noa 1.5 MB/s | 19 kB 00:00 2023-09-30T00:14:18.329 INFO:teuthology.orchestra.run.smithi121.stdout:(17/28): python3-singledispatch-3.4.0.3-18.el8. 2.0 MB/s | 24 kB 00:00 2023-09-30T00:14:18.335 INFO:teuthology.orchestra.run.smithi121.stdout:(18/28): python3-tempora-1.14.1-5.el8.noarch.rp 2.3 MB/s | 29 kB 00:00 2023-09-30T00:14:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:(19/28): python3-trustme-0.6.0-4.el8.noarch.rpm 2.2 MB/s | 27 kB 00:00 2023-09-30T00:14:18.351 INFO:teuthology.orchestra.run.smithi121.stdout:(20/28): python3-waitress-1.4.3-1.el8.noarch.rp 16 MB/s | 255 kB 00:00 2023-09-30T00:14:18.364 INFO:teuthology.orchestra.run.smithi121.stdout:(21/28): python3-webtest-2.0.33-1.el8.noarch.rp 6.3 MB/s | 85 kB 00:00 2023-09-30T00:14:18.376 INFO:teuthology.orchestra.run.smithi121.stdout:(22/28): python3-zc-lockfile-2.0-2.el8.noarch.r 1.9 MB/s | 23 kB 00:00 2023-09-30T00:14:18.378 INFO:teuthology.orchestra.run.smithi121.stdout:(23/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 6.7 MB/s | 251 kB 00:00 2023-09-30T00:14:18.410 INFO:teuthology.orchestra.run.smithi121.stdout:(24/28): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86 3.6 MB/s | 3.9 MB 00:01 2023-09-30T00:14:18.414 INFO:teuthology.orchestra.run.smithi121.stdout:(25/28): python3-markupsafe-0.23-19.el8.x86_64. 1.1 MB/s | 39 kB 00:00 2023-09-30T00:14:18.416 INFO:teuthology.orchestra.run.smithi121.stdout:(26/28): python3-werkzeug-0.12.2-4.el8.noarch.r 11 MB/s | 457 kB 00:00 2023-09-30T00:14:18.430 INFO:teuthology.orchestra.run.smithi121.stdout:(27/28): python3-pytz-2017.2-9.el8.noarch.rpm 3.3 MB/s | 54 kB 00:00 2023-09-30T00:14:18.442 INFO:teuthology.orchestra.run.smithi121.stdout:(28/28): python3-mako-1.0.6-13.el8.noarch.rpm 4.8 MB/s | 157 kB 00:00 2023-09-30T00:14:18.447 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:14:18.447 INFO:teuthology.orchestra.run.smithi121.stdout:Total 20 MB/s | 27 MB 00:01 2023-09-30T00:14:18.451 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:14:18.486 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:14:18.487 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:14:18.643 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:09 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:14:18.880 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:14:18.885 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:14:19.988 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:14:20.241 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-09-30T00:14:20.496 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-09-30T00:14:20.502 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:14:20.505 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================================= 2023-09-30T00:14:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2023-09-30T00:14:20.505 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================================= 2023-09-30T00:14:20.505 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:14:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T00:14:20.505 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2023-09-30T00:14:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T00:14:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T00:14:20.505 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T00:14:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T00:14:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:14:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T00:14:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T00:14:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T00:14:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T00:14:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T00:14:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T00:14:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T00:14:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-30T00:14:20.506 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-30T00:14:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T00:14:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T00:14:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T00:14:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-30T00:14:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T00:14:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T00:14:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T00:14:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T00:14:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T00:14:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T00:14:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T00:14:20.507 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-30T00:14:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T00:14:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:20.508 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:14:20.508 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================================= 2023-09-30T00:14:20.508 INFO:teuthology.orchestra.run.smithi094.stdout:Install 28 Packages 2023-09-30T00:14:20.508 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:20.508 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 27 M 2023-09-30T00:14:20.509 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 106 M 2023-09-30T00:14:20.509 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:14:20.702 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-09-30T00:14:20.805 INFO:teuthology.orchestra.run.smithi094.stdout:(1/28): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 43 kB/s | 6.4 kB 00:00 2023-09-30T00:14:20.913 INFO:teuthology.orchestra.run.smithi094.stdout:(2/28): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 6.1 MB/s | 1.6 MB 00:00 2023-09-30T00:14:20.988 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-09-30T00:14:21.233 INFO:teuthology.orchestra.run.smithi094.stdout:(3/28): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 4.0 MB/s | 2.3 MB 00:00 2023-09-30T00:14:21.290 INFO:teuthology.orchestra.run.smithi094.stdout:(4/28): ceph-mgr-modules-core-17.2.6-1165.g93dc 4.1 MB/s | 238 kB 00:00 2023-09-30T00:14:21.303 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-09-30T00:14:21.449 INFO:teuthology.orchestra.run.smithi094.stdout:(5/28): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 6.1 MB/s | 3.9 MB 00:00 2023-09-30T00:14:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-09-30T00:14:21.595 INFO:teuthology.orchestra.run.smithi094.stdout:(6/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 143 kB/s | 44 kB 00:00 2023-09-30T00:14:21.760 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-09-30T00:14:21.834 INFO:teuthology.orchestra.run.smithi094.stdout:(7/28): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86_ 18 MB/s | 17 MB 00:00 2023-09-30T00:14:21.846 INFO:teuthology.orchestra.run.smithi094.stdout:(8/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 687 kB/s | 173 kB 00:00 2023-09-30T00:14:21.894 INFO:teuthology.orchestra.run.smithi094.stdout:(9/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 417 kB/s | 185 kB 00:00 2023-09-30T00:14:21.923 INFO:teuthology.orchestra.run.smithi094.stdout:(10/28): python3-jaraco-6.2-6.el8.noarch.rpm 145 kB/s | 11 kB 00:00 2023-09-30T00:14:21.947 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-09-30T00:14:21.975 INFO:teuthology.orchestra.run.smithi094.stdout:(11/28): python3-jaraco-functools-2.0-4.el8.noa 218 kB/s | 18 kB 00:00 2023-09-30T00:14:22.011 INFO:teuthology.orchestra.run.smithi094.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 556 kB/s | 49 kB 00:00 2023-09-30T00:14:22.064 INFO:teuthology.orchestra.run.smithi094.stdout:(13/28): python3-more-itertools-7.2.0-3.el8.noa 657 kB/s | 59 kB 00:00 2023-09-30T00:14:22.121 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-09-30T00:14:22.141 INFO:teuthology.orchestra.run.smithi094.stdout:(14/28): python3-portend-2.6-1.el8.noarch.rpm 204 kB/s | 16 kB 00:00 2023-09-30T00:14:22.169 INFO:teuthology.orchestra.run.smithi094.stdout:(15/28): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2023-09-30T00:14:22.218 INFO:teuthology.orchestra.run.smithi094.stdout:(16/28): python3-simplegeneric-0.8.1-17.el8.noa 242 kB/s | 19 kB 00:00 2023-09-30T00:14:22.244 INFO:teuthology.orchestra.run.smithi094.stdout:(17/28): python3-singledispatch-3.4.0.3-18.el8. 320 kB/s | 24 kB 00:00 2023-09-30T00:14:22.294 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-09-30T00:14:22.299 INFO:teuthology.orchestra.run.smithi094.stdout:(18/28): python3-tempora-1.14.1-5.el8.noarch.rp 364 kB/s | 29 kB 00:00 2023-09-30T00:14:22.320 INFO:teuthology.orchestra.run.smithi094.stdout:(19/28): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2023-09-30T00:14:22.358 INFO:teuthology.orchestra.run.smithi094.stdout:(20/28): python3-cherrypy-18.4.0-1.el8.noarch.r 734 kB/s | 384 kB 00:00 2023-09-30T00:14:22.421 INFO:teuthology.orchestra.run.smithi094.stdout:(21/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.4 MB/s | 251 kB 00:00 2023-09-30T00:14:22.443 INFO:teuthology.orchestra.run.smithi094.stdout:(22/28): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2023-09-30T00:14:22.448 INFO:teuthology.orchestra.run.smithi094.stdout:(23/28): python3-waitress-1.4.3-1.el8.noarch.rp 1.7 MB/s | 255 kB 00:00 2023-09-30T00:14:22.466 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-09-30T00:14:22.478 INFO:teuthology.orchestra.run.smithi094.stdout:(24/28): python3-werkzeug-0.12.2-4.el8.noarch.r 13 MB/s | 457 kB 00:00 2023-09-30T00:14:22.490 INFO:teuthology.orchestra.run.smithi094.stdout:(25/28): python3-markupsafe-0.23-19.el8.x86_64. 933 kB/s | 39 kB 00:00 2023-09-30T00:14:22.494 INFO:teuthology.orchestra.run.smithi094.stdout:(26/28): python3-mako-1.0.6-13.el8.noarch.rpm 9.8 MB/s | 157 kB 00:00 2023-09-30T00:14:22.495 INFO:teuthology.orchestra.run.smithi094.stdout:(27/28): python3-zc-lockfile-2.0-2.el8.noarch.r 307 kB/s | 23 kB 00:00 2023-09-30T00:14:22.505 INFO:teuthology.orchestra.run.smithi094.stdout:(28/28): python3-pytz-2017.2-9.el8.noarch.rpm 3.4 MB/s | 54 kB 00:00 2023-09-30T00:14:22.508 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:14:22.509 INFO:teuthology.orchestra.run.smithi094.stdout:Total 14 MB/s | 27 MB 00:02 2023-09-30T00:14:22.513 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:14:22.549 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:14:22.549 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:14:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-09-30T00:14:22.853 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-09-30T00:14:22.941 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:14:22.945 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:14:23.071 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-09-30T00:14:23.268 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-09-30T00:14:23.470 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-09-30T00:14:23.748 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-09-30T00:14:23.995 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-09-30T00:14:24.147 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:14:24.224 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-09-30T00:14:24.410 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-09-30T00:14:24.506 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-09-30T00:14:24.654 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-09-30T00:14:24.797 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-09-30T00:14:24.860 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-09-30T00:14:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-09-30T00:14:25.113 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-09-30T00:14:25.452 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-09-30T00:14:25.470 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 23/28 2023-09-30T00:14:25.715 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-09-30T00:14:25.971 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-09-30T00:14:26.206 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-09-30T00:14:26.405 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-09-30T00:14:26.456 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T00:14:26.552 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-09-30T00:14:26.759 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-09-30T00:14:26.967 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-09-30T00:14:27.095 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-09-30T00:14:27.320 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-09-30T00:14:27.542 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-09-30T00:14:27.745 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-09-30T00:14:27.944 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T00:14:27.985 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-09-30T00:14:28.114 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T00:14:28.278 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-09-30T00:14:28.508 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T00:14:28.525 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-09-30T00:14:28.663 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T00:14:28.847 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-09-30T00:14:28.888 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T00:14:29.021 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T00:14:29.040 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T00:14:29.179 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T00:14:29.587 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T00:14:29.587 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/28 2023-09-30T00:14:29.587 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/28 2023-09-30T00:14:29.587 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/28 2023-09-30T00:14:29.587 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/28 2023-09-30T00:14:29.587 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/28 2023-09-30T00:14:29.588 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/28 2023-09-30T00:14:29.588 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-09-30T00:14:29.588 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-09-30T00:14:29.588 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-09-30T00:14:29.588 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-09-30T00:14:29.588 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-09-30T00:14:29.588 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-09-30T00:14:29.588 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-09-30T00:14:29.588 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-09-30T00:14:29.588 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-09-30T00:14:29.588 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-09-30T00:14:29.589 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-09-30T00:14:29.589 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-09-30T00:14:29.589 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-09-30T00:14:29.589 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-09-30T00:14:29.589 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-09-30T00:14:29.589 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-09-30T00:14:29.589 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-09-30T00:14:29.589 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-09-30T00:14:29.589 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-09-30T00:14:29.589 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-09-30T00:14:29.590 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-09-30T00:14:29.763 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-09-30T00:14:30.077 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-09-30T00:14:30.418 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 23/28 2023-09-30T00:14:30.645 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T00:14:31.201 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-09-30T00:14:31.202 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:25 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:14:32.135 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T00:14:32.320 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T00:14:32.716 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T00:14:32.906 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T00:14:33.033 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:14:33.132 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:33.133 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:14:33.133 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:33.133 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:33.133 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:33.133 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:14:33.133 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:33.133 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:33.133 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T00:14:33.133 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T00:14:33.133 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T00:14:33.134 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T00:14:33.134 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T00:14:33.134 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T00:14:33.134 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T00:14:33.134 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-30T00:14:33.134 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T00:14:33.134 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T00:14:33.134 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T00:14:33.134 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T00:14:33.134 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-30T00:14:33.134 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T00:14:33.135 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T00:14:33.135 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T00:14:33.135 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T00:14:33.135 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T00:14:33.135 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T00:14:33.135 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T00:14:33.135 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T00:14:33.135 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T00:14:33.135 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:33.135 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:14:33.136 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T00:14:33.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2023-09-30T00:14:33.311 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T00:14:33.329 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T00:14:33.446 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T00:14:33.644 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:14:33.833 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T00:14:33.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/28 2023-09-30T00:14:33.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/28 2023-09-30T00:14:33.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/28 2023-09-30T00:14:33.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/28 2023-09-30T00:14:33.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/28 2023-09-30T00:14:33.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/28 2023-09-30T00:14:33.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-09-30T00:14:33.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-09-30T00:14:33.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-09-30T00:14:33.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-09-30T00:14:33.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-09-30T00:14:33.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-09-30T00:14:33.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-09-30T00:14:33.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-09-30T00:14:33.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-09-30T00:14:33.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-09-30T00:14:33.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-09-30T00:14:33.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-09-30T00:14:33.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-09-30T00:14:33.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-09-30T00:14:33.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-09-30T00:14:33.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-09-30T00:14:33.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-09-30T00:14:33.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-09-30T00:14:33.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-09-30T00:14:33.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-09-30T00:14:33.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-09-30T00:14:34.485 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:28 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:14:35.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-09-30T00:14:35.427 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:26 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:14:35.793 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:14:36.269 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:14:36.270 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-30T00:14:36.270 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:14:36.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2023-09-30T00:14:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:14:37.325 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:14:37.424 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:37.424 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:14:37.425 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:37.425 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:37.425 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:37.425 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:14:37.425 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:37.425 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:37.425 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T00:14:37.425 INFO:teuthology.orchestra.run.smithi094.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T00:14:37.425 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T00:14:37.425 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T00:14:37.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T00:14:37.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T00:14:37.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T00:14:37.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-30T00:14:37.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T00:14:37.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T00:14:37.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T00:14:37.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T00:14:37.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-30T00:14:37.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T00:14:37.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T00:14:37.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T00:14:37.427 INFO:teuthology.orchestra.run.smithi094.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T00:14:37.427 INFO:teuthology.orchestra.run.smithi094.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T00:14:37.427 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T00:14:37.427 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T00:14:37.427 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T00:14:37.427 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T00:14:37.427 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:37.427 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:14:37.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base 2023-09-30T00:14:37.623 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:31 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:14:37.933 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:14:38.854 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:29 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:14:39.474 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:14:39.475 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:14:39.475 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-30T00:14:39.475 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:14:39.475 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:14:39.475 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T00:14:39.476 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:39.476 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:14:39.476 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:14:39.476 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-30T00:14:39.476 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:39.476 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 80 k 2023-09-30T00:14:39.476 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 363 k 2023-09-30T00:14:39.476 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:14:39.609 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 606 kB/s | 80 kB 00:00 2023-09-30T00:14:39.609 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:14:39.609 INFO:teuthology.orchestra.run.smithi121.stdout:Total 600 kB/s | 80 kB 00:00 2023-09-30T00:14:39.609 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:14:39.617 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:14:39.617 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:14:39.619 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:14:39.624 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:14:39.679 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:14:40.847 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:14:41.338 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:14:41.339 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2023-09-30T00:14:41.339 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:14:41.725 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install cephadm 2023-09-30T00:14:42.085 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:14:42.484 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:14:42.724 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:14:42.927 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:14:42.933 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:33 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:14:44.739 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:14:44.739 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:38 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:14:44.837 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:14:44.838 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:14:44.838 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2023-09-30T00:14:44.839 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:14:44.839 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:14:44.839 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T00:14:44.839 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:44.839 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:14:44.839 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:14:44.839 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2023-09-30T00:14:44.839 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:44.839 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 80 k 2023-09-30T00:14:44.839 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 363 k 2023-09-30T00:14:44.839 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:14:45.061 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 361 kB/s | 80 kB 00:00 2023-09-30T00:14:45.061 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:14:45.062 INFO:teuthology.orchestra.run.smithi094.stdout:Total 359 kB/s | 80 kB 00:00 2023-09-30T00:14:45.062 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:14:45.070 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:14:45.070 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:14:45.072 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:14:45.076 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:14:45.122 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:14:46.659 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:14:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:46.755 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:14:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:14:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:46.755 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:14:46.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T00:14:47.272 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:14:48.103 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:42 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:14:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:14:48.688 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:14:48.889 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:14:49.869 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:14:49.869 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:14:49.930 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:14:49.931 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:14:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2023-09-30T00:14:49.931 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:14:49.931 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:14:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T00:14:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:49.931 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:14:49.931 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:14:49.931 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-30T00:14:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:49.932 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 161 k 2023-09-30T00:14:49.932 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 490 k 2023-09-30T00:14:49.932 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:14:50.189 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 628 kB/s | 161 kB 00:00 2023-09-30T00:14:50.190 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:14:50.190 INFO:teuthology.orchestra.run.smithi121.stdout:Total 625 kB/s | 161 kB 00:00 2023-09-30T00:14:50.190 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:14:50.202 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:14:50.203 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:14:50.256 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:14:50.260 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:14:50.372 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:14:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:14:50.936 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:14:51.808 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:14:51.903 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:51.903 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:14:51.904 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:14:51.904 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:51.904 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:14:51.914 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:14:51.915 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:45 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:14:52.076 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T00:14:52.453 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:14:53.285 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:44 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:14:53.776 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:14:53.872 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:53.872 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:14:53.872 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:53.872 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:14:53.872 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:14:54.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2023-09-30T00:14:54.381 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:14:55.066 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:14:55.067 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:14:55.067 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2023-09-30T00:14:55.067 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:14:55.067 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:14:55.067 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T00:14:55.068 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:55.068 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:14:55.068 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:14:55.068 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2023-09-30T00:14:55.068 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:55.068 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 161 k 2023-09-30T00:14:55.068 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 490 k 2023-09-30T00:14:55.068 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:14:55.275 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:49 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:14:55.327 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 623 kB/s | 161 kB 00:00 2023-09-30T00:14:55.327 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:14:55.328 INFO:teuthology.orchestra.run.smithi094.stdout:Total 620 kB/s | 161 kB 00:00 2023-09-30T00:14:55.328 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:14:55.340 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:14:55.340 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:14:55.393 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:14:55.397 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:14:55.549 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:14:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:14:56.164 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:14:56.628 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:14:57.103 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:14:57.104 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-30T00:14:57.104 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:14:57.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T00:14:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:14:57.259 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:48 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:14:57.574 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:14:58.411 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:52 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:14:59.149 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:14:59.246 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:59.246 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:14:59.246 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:14:59.246 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:14:59.246 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:14:59.375 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr 2023-09-30T00:14:59.739 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:15:00.223 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:15:00.250 INFO:teuthology.orchestra.run.smithi121.stdout:=================================================================================================== 2023-09-30T00:15:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-30T00:15:00.250 INFO:teuthology.orchestra.run.smithi121.stdout:=================================================================================================== 2023-09-30T00:15:00.250 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:15:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T00:15:00.250 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-30T00:15:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T00:15:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T00:15:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T00:15:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T00:15:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T00:15:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:00.251 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:15:00.251 INFO:teuthology.orchestra.run.smithi121.stdout:=================================================================================================== 2023-09-30T00:15:00.251 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2023-09-30T00:15:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:00.251 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 2.0 M 2023-09-30T00:15:00.252 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 8.0 M 2023-09-30T00:15:00.252 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:15:00.681 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 172 kB/s | 23 kB 00:00 2023-09-30T00:15:00.719 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:51 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:15:00.776 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 63 kB/s | 14 kB 00:00 2023-09-30T00:15:00.844 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 207 kB/s | 34 kB 00:00 2023-09-30T00:15:00.892 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 902 kB/s | 43 kB 00:00 2023-09-30T00:15:00.951 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2023-09-30T00:15:00.982 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 3.9 MB/s | 1.7 MB 00:00 2023-09-30T00:15:00.984 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:00.985 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2023-09-30T00:15:00.987 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:15:00.996 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:15:00.996 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:15:01.078 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:15:01.082 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:15:01.474 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:15:01.693 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-09-30T00:15:01.915 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-09-30T00:15:02.064 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:15:02.108 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-09-30T00:15:02.245 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T00:15:02.537 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:15:02.538 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2023-09-30T00:15:02.539 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:15:02.646 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T00:15:02.661 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T00:15:02.906 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:15:03.009 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:15:03.164 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T00:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T00:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T00:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-09-30T00:15:03.165 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-09-30T00:15:03.867 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:54 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:15:04.157 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-09-30T00:15:04.157 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:58 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:15:05.690 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:15:05.690 INFO:teuthology.orchestra.run.smithi094.stdout:=================================================================================================== 2023-09-30T00:15:05.691 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2023-09-30T00:15:05.691 INFO:teuthology.orchestra.run.smithi094.stdout:=================================================================================================== 2023-09-30T00:15:05.691 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:15:05.691 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T00:15:05.691 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2023-09-30T00:15:05.691 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T00:15:05.691 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T00:15:05.691 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T00:15:05.691 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T00:15:05.691 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T00:15:05.692 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:05.692 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:15:05.692 INFO:teuthology.orchestra.run.smithi094.stdout:=================================================================================================== 2023-09-30T00:15:05.692 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2023-09-30T00:15:05.692 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:05.692 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 2.0 M 2023-09-30T00:15:05.693 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 8.0 M 2023-09-30T00:15:05.693 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:15:05.946 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 126 kB/s | 14 kB 00:00 2023-09-30T00:15:05.997 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:15:06.016 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): ceph-grafana-dashboards-17.2.6-1165.g93d 125 kB/s | 23 kB 00:00 2023-09-30T00:15:06.072 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 7.1 MB/s | 1.7 MB 00:00 2023-09-30T00:15:06.095 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:06.095 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:15:06.095 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:06.095 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:06.095 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:06.095 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T00:15:06.095 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T00:15:06.095 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T00:15:06.095 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:06.096 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:15:06.100 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.5 MB/s | 43 kB 00:00 2023-09-30T00:15:06.118 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 196 kB/s | 34 kB 00:00 2023-09-30T00:15:06.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T00:15:06.279 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 749 kB/s | 196 kB 00:00 2023-09-30T00:15:06.282 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:06.283 INFO:teuthology.orchestra.run.smithi094.stdout:Total 3.4 MB/s | 2.0 MB 00:00 2023-09-30T00:15:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:15:06.292 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:15:06.292 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:15:06.375 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:15:06.379 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:15:06.579 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:15:06.655 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:15:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-09-30T00:15:07.102 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-09-30T00:15:07.295 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-09-30T00:15:07.428 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:01 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:15:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T00:15:07.871 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T00:15:08.131 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:15:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:15:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T00:15:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T00:15:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T00:15:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-09-30T00:15:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-09-30T00:15:09.267 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:15:09.269 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================================================== 2023-09-30T00:15:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-30T00:15:09.269 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================================================== 2023-09-30T00:15:09.269 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:15:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T00:15:09.269 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-30T00:15:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-09-30T00:15:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-09-30T00:15:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-09-30T00:15:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-30T00:15:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-30T00:15:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-30T00:15:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-09-30T00:15:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:09.270 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:15:09.270 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================================================== 2023-09-30T00:15:09.271 INFO:teuthology.orchestra.run.smithi121.stdout:Install 8 Packages 2023-09-30T00:15:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:09.271 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 36 M 2023-09-30T00:15:09.271 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 199 M 2023-09-30T00:15:09.272 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:15:09.302 INFO:teuthology.orchestra.run.smithi121.stdout:(1/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 7.0 MB/s | 225 kB 00:00 2023-09-30T00:15:09.371 INFO:teuthology.orchestra.run.smithi121.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 55 MB/s | 3.7 MB 00:00 2023-09-30T00:15:09.384 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-09-30T00:15:09.384 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:00 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:15:09.454 INFO:teuthology.orchestra.run.smithi121.stdout:(3/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 58 MB/s | 4.7 MB 00:00 2023-09-30T00:15:09.581 INFO:teuthology.orchestra.run.smithi121.stdout:(4/8): python3-scipy-1.0.0-20.module+el8.1.0+33 47 MB/s | 15 MB 00:00 2023-09-30T00:15:09.597 INFO:teuthology.orchestra.run.smithi121.stdout:(5/8): libquadmath-8.4.1-1.el8.x86_64.rpm 10 MB/s | 169 kB 00:00 2023-09-30T00:15:09.620 INFO:teuthology.orchestra.run.smithi121.stdout:(6/8): libgfortran-8.4.1-1.el8.x86_64.rpm 28 MB/s | 639 kB 00:00 2023-09-30T00:15:09.636 INFO:teuthology.orchestra.run.smithi121.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 25 MB/s | 4.6 MB 00:00 2023-09-30T00:15:09.745 INFO:teuthology.orchestra.run.smithi121.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 16 MB/s | 7.4 MB 00:00 2023-09-30T00:15:09.747 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:09.747 INFO:teuthology.orchestra.run.smithi121.stdout:Total 75 MB/s | 36 MB 00:00 2023-09-30T00:15:09.756 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:15:09.775 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:15:09.775 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:15:10.198 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:15:10.203 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:15:10.836 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:15:11.020 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T00:15:11.164 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T00:15:11.189 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:15:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:11.287 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:15:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T00:15:11.287 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T00:15:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T00:15:11.288 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:11.288 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:15:11.328 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:15:11.404 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T00:15:11.767 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:15:11.905 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:15:12.103 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T00:15:12.600 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:03 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:15:12.639 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T00:15:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T00:15:13.579 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T00:15:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-09-30T00:15:14.153 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:15:14.397 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:15:14.398 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================================================== 2023-09-30T00:15:14.399 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2023-09-30T00:15:14.399 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================================================== 2023-09-30T00:15:14.399 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:15:14.399 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T00:15:14.399 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2023-09-30T00:15:14.399 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-09-30T00:15:14.399 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-09-30T00:15:14.399 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-09-30T00:15:14.399 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-30T00:15:14.399 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-30T00:15:14.400 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-30T00:15:14.400 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-09-30T00:15:14.400 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:14.400 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:15:14.400 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================================================== 2023-09-30T00:15:14.400 INFO:teuthology.orchestra.run.smithi094.stdout:Install 8 Packages 2023-09-30T00:15:14.400 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:14.401 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 36 M 2023-09-30T00:15:14.401 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 199 M 2023-09-30T00:15:14.401 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:15:14.442 INFO:teuthology.orchestra.run.smithi094.stdout:(1/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 5.4 MB/s | 225 kB 00:00 2023-09-30T00:15:14.507 INFO:teuthology.orchestra.run.smithi094.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 57 MB/s | 3.7 MB 00:00 2023-09-30T00:15:14.590 INFO:teuthology.orchestra.run.smithi094.stdout:(3/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 58 MB/s | 4.7 MB 00:00 2023-09-30T00:15:14.733 INFO:teuthology.orchestra.run.smithi094.stdout:(4/8): python3-scipy-1.0.0-20.module+el8.1.0+33 44 MB/s | 15 MB 00:00 2023-09-30T00:15:14.753 INFO:teuthology.orchestra.run.smithi094.stdout:(5/8): libquadmath-8.4.1-1.el8.x86_64.rpm 8.3 MB/s | 169 kB 00:00 2023-09-30T00:15:14.779 INFO:teuthology.orchestra.run.smithi094.stdout:(6/8): libgfortran-8.4.1-1.el8.x86_64.rpm 24 MB/s | 639 kB 00:00 2023-09-30T00:15:14.794 INFO:teuthology.orchestra.run.smithi094.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 22 MB/s | 4.6 MB 00:00 2023-09-30T00:15:14.904 INFO:teuthology.orchestra.run.smithi094.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 15 MB/s | 7.4 MB 00:00 2023-09-30T00:15:14.905 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:14.906 INFO:teuthology.orchestra.run.smithi094.stdout:Total 71 MB/s | 36 MB 00:00 2023-09-30T00:15:14.915 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:15:14.934 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:15:14.934 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:15:15.351 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:15:15.355 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:15:16.031 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:15:16.234 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:15:16.241 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T00:15:16.378 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T00:15:16.524 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:15:17.062 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T00:15:17.249 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T00:15:17.702 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-09-30T00:15:17.778 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T00:15:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:15:17.974 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T00:15:18.211 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:15:18.211 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T00:15:18.211 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 2/8 2023-09-30T00:15:18.211 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 3/8 2023-09-30T00:15:18.212 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2023-09-30T00:15:18.212 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 5/8 2023-09-30T00:15:18.212 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 6/8 2023-09-30T00:15:18.212 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2023-09-30T00:15:18.715 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T00:15:19.075 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-09-30T00:15:19.231 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2023-09-30T00:15:19.231 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:13 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:15:19.307 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:15:21.202 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:15:21.301 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:21.301 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:15:21.301 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:21.301 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-09-30T00:15:21.301 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-09-30T00:15:21.301 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.12-1.el8.x86_64 2023-09-30T00:15:21.302 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-09-30T00:15:21.302 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-09-30T00:15:21.302 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-09-30T00:15:21.302 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-09-30T00:15:21.302 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:21.302 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:15:21.395 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T00:15:21.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2023-09-30T00:15:21.806 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:15:22.631 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:16 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:15:22.664 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-09-30T00:15:22.840 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:15:23.154 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:15:23.154 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T00:15:23.154 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 2/8 2023-09-30T00:15:23.154 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 3/8 2023-09-30T00:15:23.154 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2023-09-30T00:15:23.154 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 5/8 2023-09-30T00:15:23.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 6/8 2023-09-30T00:15:23.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2023-09-30T00:15:24.193 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2023-09-30T00:15:24.193 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:15 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:15:24.473 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:15:24.475 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================================== 2023-09-30T00:15:24.475 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-30T00:15:24.475 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================================== 2023-09-30T00:15:24.475 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:15:24.475 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T00:15:24.475 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-30T00:15:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T00:15:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T00:15:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T00:15:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-30T00:15:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-30T00:15:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T00:15:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-30T00:15:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-30T00:15:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-30T00:15:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T00:15:24.476 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T00:15:24.477 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T00:15:24.477 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:24.477 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:15:24.477 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================================== 2023-09-30T00:15:24.477 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2023-09-30T00:15:24.477 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:24.478 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 1.9 M 2023-09-30T00:15:24.478 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 25 M 2023-09-30T00:15:24.478 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:15:24.761 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 358 kB/s | 47 kB 00:00 2023-09-30T00:15:24.849 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 74 kB/s | 16 kB 00:00 2023-09-30T00:15:24.864 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): python3-cachetools-3.1.1-4.el8.noarch.r 140 kB/s | 33 kB 00:00 2023-09-30T00:15:24.979 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 556 kB/s | 64 kB 00:00 2023-09-30T00:15:25.048 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 896 kB/s | 61 kB 00:00 2023-09-30T00:15:25.054 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-google-auth-1.1.1-10.el8.noarch 277 kB/s | 81 kB 00:00 2023-09-30T00:15:25.079 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 854 kB/s | 27 kB 00:00 2023-09-30T00:15:25.086 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.8 MB/s | 126 kB 00:00 2023-09-30T00:15:25.094 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-pyasn1-modules-0.3.7-6.el8.noar 7.3 MB/s | 110 kB 00:00 2023-09-30T00:15:25.103 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): python3-jsonpointer-1.10-11.el8.noarch 1.2 MB/s | 20 kB 00:00 2023-09-30T00:15:25.110 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 9.4 MB/s | 155 kB 00:00 2023-09-30T00:15:25.120 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-requests-oauthlib-1.0.0-1.el8. 2.4 MB/s | 43 kB 00:00 2023-09-30T00:15:25.179 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.5 MB/s | 1.2 MB 00:00 2023-09-30T00:15:25.181 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:25.182 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.7 MB/s | 1.9 MB 00:00 2023-09-30T00:15:25.187 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:15:25.201 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:15:25.201 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:15:25.289 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:15:25.293 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:15:25.804 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:15:26.115 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T00:15:26.263 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:15:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:26.366 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:15:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:26.366 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-09-30T00:15:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-09-30T00:15:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.12-1.el8.x86_64 2023-09-30T00:15:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-09-30T00:15:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-09-30T00:15:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-09-30T00:15:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-09-30T00:15:26.367 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:26.367 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:15:26.368 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-09-30T00:15:26.532 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-rook 2023-09-30T00:15:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-09-30T00:15:26.899 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:15:26.965 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2023-09-30T00:15:27.557 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2023-09-30T00:15:27.782 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2023-09-30T00:15:27.795 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:18 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:15:28.009 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2023-09-30T00:15:28.274 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-09-30T00:15:28.499 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T00:15:28.758 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-09-30T00:15:29.563 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:15:29.565 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================================== 2023-09-30T00:15:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2023-09-30T00:15:29.565 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================================== 2023-09-30T00:15:29.565 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:15:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T00:15:29.565 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2023-09-30T00:15:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T00:15:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T00:15:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T00:15:29.565 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-30T00:15:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-30T00:15:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T00:15:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-30T00:15:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-30T00:15:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-30T00:15:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T00:15:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T00:15:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T00:15:29.566 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:29.566 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:15:29.566 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================================== 2023-09-30T00:15:29.566 INFO:teuthology.orchestra.run.smithi094.stdout:Install 13 Packages 2023-09-30T00:15:29.567 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:29.567 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 1.9 M 2023-09-30T00:15:29.567 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 25 M 2023-09-30T00:15:29.567 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:15:29.934 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-09-30T00:15:30.114 INFO:teuthology.orchestra.run.smithi094.stdout:(1/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 214 kB/s | 47 kB 00:00 2023-09-30T00:15:30.234 INFO:teuthology.orchestra.run.smithi094.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 48 kB/s | 16 kB 00:00 2023-09-30T00:15:30.272 INFO:teuthology.orchestra.run.smithi094.stdout:(3/13): python3-cachetools-3.1.1-4.el8.noarch.r 87 kB/s | 33 kB 00:00 2023-09-30T00:15:30.276 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-09-30T00:15:30.395 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:15:30.451 INFO:teuthology.orchestra.run.smithi094.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 358 kB/s | 64 kB 00:00 2023-09-30T00:15:30.548 INFO:teuthology.orchestra.run.smithi094.stdout:(5/13): python3-google-auth-1.1.1-10.el8.noarch 188 kB/s | 81 kB 00:00 2023-09-30T00:15:30.562 INFO:teuthology.orchestra.run.smithi094.stdout:(6/13): python3-websocket-client-0.56.0-5.el8.n 552 kB/s | 61 kB 00:00 2023-09-30T00:15:30.588 INFO:teuthology.orchestra.run.smithi094.stdout:(7/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 664 kB/s | 27 kB 00:00 2023-09-30T00:15:30.606 INFO:teuthology.orchestra.run.smithi094.stdout:(8/13): python3-pyasn1-modules-0.3.7-6.el8.noar 6.0 MB/s | 110 kB 00:00 2023-09-30T00:15:30.607 INFO:teuthology.orchestra.run.smithi094.stdout:(9/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.2 MB/s | 126 kB 00:00 2023-09-30T00:15:30.622 INFO:teuthology.orchestra.run.smithi094.stdout:(10/13): python3-jsonpointer-1.10-11.el8.noarch 1.3 MB/s | 20 kB 00:00 2023-09-30T00:15:30.625 INFO:teuthology.orchestra.run.smithi094.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 8.6 MB/s | 155 kB 00:00 2023-09-30T00:15:30.636 INFO:teuthology.orchestra.run.smithi094.stdout:(12/13): python3-requests-oauthlib-1.0.0-1.el8. 3.2 MB/s | 43 kB 00:00 2023-09-30T00:15:30.648 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:15:30.648 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T00:15:30.649 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-09-30T00:15:30.649 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-09-30T00:15:30.649 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-09-30T00:15:30.649 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-09-30T00:15:30.649 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-09-30T00:15:30.649 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-09-30T00:15:30.650 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2023-09-30T00:15:30.650 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 9/13 2023-09-30T00:15:30.650 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-09-30T00:15:30.650 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T00:15:30.650 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2023-09-30T00:15:30.735 INFO:teuthology.orchestra.run.smithi094.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.3 MB/s | 1.2 MB 00:00 2023-09-30T00:15:30.739 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:30.739 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.6 MB/s | 1.9 MB 00:01 2023-09-30T00:15:30.746 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:15:30.761 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:15:30.762 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:15:30.851 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:15:30.855 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:15:31.387 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:15:31.666 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T00:15:31.819 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2023-09-30T00:15:31.819 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:25 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:15:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-09-30T00:15:32.242 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-09-30T00:15:32.538 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2023-09-30T00:15:32.813 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2023-09-30T00:15:33.072 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2023-09-30T00:15:33.304 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2023-09-30T00:15:33.564 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-09-30T00:15:33.765 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:15:33.789 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T00:15:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:33.867 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:15:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T00:15:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T00:15:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T00:15:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T00:15:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T00:15:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T00:15:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T00:15:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T00:15:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T00:15:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T00:15:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T00:15:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T00:15:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:33.869 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:15:33.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T00:15:34.085 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-09-30T00:15:34.367 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:15:35.235 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-09-30T00:15:35.253 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:29 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:15:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-09-30T00:15:35.663 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:15:35.976 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:15:35.976 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T00:15:35.976 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-09-30T00:15:35.977 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-09-30T00:15:35.977 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-09-30T00:15:35.977 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-09-30T00:15:35.977 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-09-30T00:15:35.977 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-09-30T00:15:35.977 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2023-09-30T00:15:35.977 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 9/13 2023-09-30T00:15:35.977 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-09-30T00:15:35.977 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T00:15:35.977 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2023-09-30T00:15:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2023-09-30T00:15:37.081 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:28 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:15:37.142 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:15:37.142 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================================== 2023-09-30T00:15:37.142 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-30T00:15:37.142 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================================== 2023-09-30T00:15:37.142 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:15:37.143 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T00:15:37.143 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-30T00:15:37.143 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T00:15:37.143 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-30T00:15:37.143 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-30T00:15:37.143 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T00:15:37.143 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:37.143 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:15:37.143 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================================== 2023-09-30T00:15:37.143 INFO:teuthology.orchestra.run.smithi121.stdout:Install 5 Packages 2023-09-30T00:15:37.143 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:37.144 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 6.0 M 2023-09-30T00:15:37.144 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2023-09-30T00:15:37.144 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:15:37.395 INFO:teuthology.orchestra.run.smithi121.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 291 kB/s | 60 kB 00:00 2023-09-30T00:15:37.450 INFO:teuthology.orchestra.run.smithi121.stdout:(2/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 463 kB/s | 121 kB 00:00 2023-09-30T00:15:37.481 INFO:teuthology.orchestra.run.smithi121.stdout:(3/5): python3-babel-2.5.1-5.el8.noarch.rpm 56 MB/s | 4.8 MB 00:00 2023-09-30T00:15:37.486 INFO:teuthology.orchestra.run.smithi121.stdout:(4/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 15 MB/s | 538 kB 00:00 2023-09-30T00:15:37.573 INFO:teuthology.orchestra.run.smithi121.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.2 MB/s | 476 kB 00:00 2023-09-30T00:15:37.575 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:37.575 INFO:teuthology.orchestra.run.smithi121.stdout:Total 14 MB/s | 6.0 MB 00:00 2023-09-30T00:15:37.579 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:15:37.589 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:15:37.589 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:15:37.685 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:15:37.689 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:15:38.767 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:15:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-09-30T00:15:39.215 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:15:39.315 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:39.315 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:15:39.315 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:39.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T00:15:39.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T00:15:39.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T00:15:39.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T00:15:39.315 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T00:15:39.316 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T00:15:39.316 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T00:15:39.316 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T00:15:39.316 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T00:15:39.316 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T00:15:39.316 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T00:15:39.316 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T00:15:39.316 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:39.316 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:15:39.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T00:15:39.459 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-09-30T00:15:39.821 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:15:39.837 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T00:15:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T00:15:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:15:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:15:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T00:15:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T00:15:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T00:15:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-09-30T00:15:40.719 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:31 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:15:41.639 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-09-30T00:15:41.639 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:35 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:15:42.553 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:15:42.554 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================================== 2023-09-30T00:15:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2023-09-30T00:15:42.554 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================================== 2023-09-30T00:15:42.554 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:15:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T00:15:42.554 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2023-09-30T00:15:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T00:15:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-30T00:15:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-30T00:15:42.554 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T00:15:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:42.555 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:15:42.555 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================================== 2023-09-30T00:15:42.555 INFO:teuthology.orchestra.run.smithi094.stdout:Install 5 Packages 2023-09-30T00:15:42.555 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:42.555 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 6.0 M 2023-09-30T00:15:42.556 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 26 M 2023-09-30T00:15:42.556 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:15:42.714 INFO:teuthology.orchestra.run.smithi094.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 474 kB/s | 60 kB 00:00 2023-09-30T00:15:42.739 INFO:teuthology.orchestra.run.smithi094.stdout:(2/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 792 kB/s | 121 kB 00:00 2023-09-30T00:15:42.764 INFO:teuthology.orchestra.run.smithi094.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.6 MB/s | 476 kB 00:00 2023-09-30T00:15:42.783 INFO:teuthology.orchestra.run.smithi094.stdout:(4/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 12 MB/s | 538 kB 00:00 2023-09-30T00:15:42.802 INFO:teuthology.orchestra.run.smithi094.stdout:(5/5): python3-babel-2.5.1-5.el8.noarch.rpm 54 MB/s | 4.8 MB 00:00 2023-09-30T00:15:42.804 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:42.804 INFO:teuthology.orchestra.run.smithi094.stdout:Total 24 MB/s | 6.0 MB 00:00 2023-09-30T00:15:42.807 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:15:42.817 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:15:42.817 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:15:42.915 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:15:42.919 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:15:43.785 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:15:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:43.884 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:15:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T00:15:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.5.1-5.el8.noarch 2023-09-30T00:15:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-09-30T00:15:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T00:15:43.884 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:43.884 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:15:43.956 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:15:44.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2023-09-30T00:15:44.360 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-09-30T00:15:44.379 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:15:44.625 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-09-30T00:15:44.987 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T00:15:45.284 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:39 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:15:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T00:15:45.487 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:15:45.757 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:15:45.757 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T00:15:45.757 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T00:15:45.757 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T00:15:45.757 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-09-30T00:15:46.778 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-09-30T00:15:46.779 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:37 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:15:47.104 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:15:47.104 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:15:47.105 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-30T00:15:47.105 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:15:47.105 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:15:47.105 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse 2023-09-30T00:15:47.105 INFO:teuthology.orchestra.run.smithi121.stdout: x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T00:15:47.105 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-09-30T00:15:47.105 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-30T00:15:47.106 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:47.106 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:15:47.106 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:15:47.106 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2023-09-30T00:15:47.106 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:47.107 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 957 k 2023-09-30T00:15:47.107 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 3.0 M 2023-09-30T00:15:47.107 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:15:47.265 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 519 kB/s | 83 kB 00:00 2023-09-30T00:15:47.483 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 2.3 MB/s | 874 kB 00:00 2023-09-30T00:15:47.483 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:47.484 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.5 MB/s | 957 kB 00:00 2023-09-30T00:15:47.485 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:15:47.506 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:15:47.506 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:15:47.588 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:15:47.592 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:15:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:15:48.055 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-09-30T00:15:48.253 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:15:48.626 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:15:48.627 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:15:50.263 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:15:50.361 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:50.361 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:15:50.361 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:50.361 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T00:15:50.361 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.5.1-5.el8.noarch 2023-09-30T00:15:50.361 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-09-30T00:15:50.361 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T00:15:50.361 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:50.362 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:15:50.494 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-fuse 2023-09-30T00:15:50.732 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-09-30T00:15:50.732 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:44 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:15:50.864 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:15:52.622 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:43 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:15:54.444 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:15:54.444 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:15:54.444 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2023-09-30T00:15:54.444 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:15:54.445 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:15:54.445 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse 2023-09-30T00:15:54.445 INFO:teuthology.orchestra.run.smithi094.stdout: x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T00:15:54.445 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2023-09-30T00:15:54.445 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-30T00:15:54.445 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:54.445 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:15:54.445 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:15:54.445 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2023-09-30T00:15:54.445 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:54.446 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 957 k 2023-09-30T00:15:54.446 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 3.0 M 2023-09-30T00:15:54.446 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:15:54.493 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.7 MB/s | 83 kB 00:00 2023-09-30T00:15:54.663 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 3.9 MB/s | 874 kB 00:00 2023-09-30T00:15:54.664 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:54.664 INFO:teuthology.orchestra.run.smithi094.stdout:Total 4.3 MB/s | 957 kB 00:00 2023-09-30T00:15:54.666 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:15:54.687 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:15:54.687 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:15:54.769 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:15:54.773 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:15:54.978 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:15:55.180 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:15:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:55.276 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:15:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-09-30T00:15:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:55.276 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:15:55.334 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-09-30T00:15:55.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-volume 2023-09-30T00:15:55.511 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:15:55.776 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:15:55.907 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:15:55.908 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:15:56.787 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:50 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:15:56.914 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-09-30T00:15:56.914 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:47 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:15:58.641 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:15:58.641 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:15:58.641 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-09-30T00:15:58.642 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:15:58.642 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:15:58.642 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T00:15:58.642 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:58.642 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:15:58.642 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:15:58.642 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-30T00:15:58.642 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:15:58.642 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 267 k 2023-09-30T00:15:58.642 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.3 M 2023-09-30T00:15:58.642 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:15:58.936 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 909 kB/s | 267 kB 00:00 2023-09-30T00:15:58.936 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:58.937 INFO:teuthology.orchestra.run.smithi121.stdout:Total 905 kB/s | 267 kB 00:00 2023-09-30T00:15:58.937 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:15:58.948 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:15:58.948 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:15:58.980 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:15:58.984 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:15:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:15:59.178 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:15:59.277 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:59.277 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:15:59.277 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-09-30T00:15:59.277 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:15:59.277 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:15:59.426 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-volume 2023-09-30T00:15:59.476 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:15:59.801 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:15:59.887 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:16:00.725 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:51 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:00.870 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:16:00.870 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:54 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:02.516 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:16:02.516 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:02.517 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2023-09-30T00:16:02.517 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:02.517 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:16:02.517 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T00:16:02.517 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:02.517 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:16:02.517 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:02.517 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2023-09-30T00:16:02.517 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:02.517 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 267 k 2023-09-30T00:16:02.517 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 1.3 M 2023-09-30T00:16:02.518 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:16:02.689 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 1.5 MB/s | 267 kB 00:00 2023-09-30T00:16:02.690 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:16:02.690 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.5 MB/s | 267 kB 00:00 2023-09-30T00:16:02.690 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:16:02.701 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:16:02.701 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:16:02.721 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:16:02.726 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:16:02.912 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:16:02.928 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:16:03.026 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:03.026 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:16:03.026 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:16:03.026 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:03.026 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:16:03.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2023-09-30T00:16:03.402 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:16:03.524 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:03.819 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:16:04.416 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:58 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:04.910 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:16:04.910 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:55 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:06.232 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:16:06.232 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:06.232 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2023-09-30T00:16:06.233 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:06.233 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:16:06.233 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T00:16:06.233 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:06.233 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:16:06.233 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:06.233 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-30T00:16:06.233 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:06.234 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 130 k 2023-09-30T00:16:06.234 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 464 k 2023-09-30T00:16:06.234 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:16:06.388 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 843 kB/s | 130 kB 00:00 2023-09-30T00:16:06.388 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:16:06.389 INFO:teuthology.orchestra.run.smithi121.stdout:Total 836 kB/s | 130 kB 00:00 2023-09-30T00:16:06.389 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:16:06.395 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:16:06.395 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:16:06.451 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:16:06.455 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:16:06.602 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:16:06.807 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:06.984 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:16:07.078 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:07.081 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:07.081 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:16:07.081 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:16:07.081 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:07.081 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:16:07.236 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados-devel 2023-09-30T00:16:07.613 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:08.083 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:08.083 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:02 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:08.534 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:59 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:10.108 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:16:10.205 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:10.205 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:16:10.205 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:16:10.205 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:10.205 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:16:10.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2023-09-30T00:16:10.409 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:16:10.410 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:10.410 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2023-09-30T00:16:10.410 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:10.410 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:16:10.410 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T00:16:10.410 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:10.410 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:16:10.410 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:10.411 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2023-09-30T00:16:10.411 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:10.411 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 130 k 2023-09-30T00:16:10.411 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 464 k 2023-09-30T00:16:10.411 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:16:10.564 INFO:teuthology.orchestra.run.smithi094.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 848 kB/s | 130 kB 00:00 2023-09-30T00:16:10.565 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:16:10.565 INFO:teuthology.orchestra.run.smithi094.stdout:Total 841 kB/s | 130 kB 00:00 2023-09-30T00:16:10.565 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:16:10.571 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:16:10.572 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:16:10.627 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:16:10.631 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:16:10.702 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:10.754 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:16:10.967 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:11.234 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:11.699 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:05 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:12.240 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:12.240 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:03 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:13.019 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:13.498 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:16:13.499 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-30T00:16:13.499 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:16:13.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2023-09-30T00:16:13.974 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:14.379 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:16:14.476 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:14.476 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:16:14.477 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:16:14.477 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:14.477 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:16:14.769 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs2 2023-09-30T00:16:14.883 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:08 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:15.143 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:16.013 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:07 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:16.668 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:16:16.669 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:16.669 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2023-09-30T00:16:16.669 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:16.669 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:16:16.669 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T00:16:16.669 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:16.669 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:16:16.669 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:16.669 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-30T00:16:16.670 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:16.670 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 25 k 2023-09-30T00:16:16.670 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 109 k 2023-09-30T00:16:16.670 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:16:16.781 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 224 kB/s | 25 kB 00:00 2023-09-30T00:16:16.781 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:16:16.782 INFO:teuthology.orchestra.run.smithi121.stdout:Total 221 kB/s | 25 kB 00:00 2023-09-30T00:16:16.782 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:16:16.786 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:16:16.786 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:16:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:16:16.792 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:16:16.878 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:16:17.047 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:17.305 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:17.322 INFO:teuthology.orchestra.run.smithi094.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:17.802 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:16:17.803 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2023-09-30T00:16:17.803 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:16:17.919 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs-devel 2023-09-30T00:16:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:18.282 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:12 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:18.289 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:19.134 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:10 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:20.169 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:16:20.266 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:20.266 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:16:20.266 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:16:20.267 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:20.267 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:16:20.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2023-09-30T00:16:20.778 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:20.973 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:16:20.974 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2023-09-30T00:16:20.974 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:20.974 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:16:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T00:16:20.974 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:20.974 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:16:20.975 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:20.975 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2023-09-30T00:16:20.975 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:20.975 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 25 k 2023-09-30T00:16:20.975 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 109 k 2023-09-30T00:16:20.975 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:16:21.161 INFO:teuthology.orchestra.run.smithi094.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 134 kB/s | 25 kB 00:00 2023-09-30T00:16:21.161 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:16:21.162 INFO:teuthology.orchestra.run.smithi094.stdout:Total 133 kB/s | 25 kB 00:00 2023-09-30T00:16:21.162 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:16:21.165 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:16:21.165 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:16:21.167 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:16:21.171 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:16:21.231 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:16:21.391 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:21.605 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:15 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:21.658 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:22.657 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:22.657 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:13 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:22.990 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:23.463 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:16:23.464 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-30T00:16:23.464 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:16:23.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2023-09-30T00:16:23.970 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:24.550 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:16:24.646 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:24.647 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:16:24.647 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:16:24.647 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:24.647 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:16:24.798 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados2 2023-09-30T00:16:24.905 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:18 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:25.167 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:26.067 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:17 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:26.244 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:26.716 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:16:26.717 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-30T00:16:26.717 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:16:26.820 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2023-09-30T00:16:27.190 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:27.462 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:27.934 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:16:27.935 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2023-09-30T00:16:27.936 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:16:28.045 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librbd1 2023-09-30T00:16:28.048 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:22 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:28.414 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:29.370 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:20 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:29.391 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:29.861 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:16:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-30T00:16:29.862 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:16:29.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2023-09-30T00:16:30.351 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:30.729 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:31.188 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:25 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:31.196 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:16:31.197 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2023-09-30T00:16:31.197 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:16:31.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rados 2023-09-30T00:16:31.677 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:32.644 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:23 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:32.791 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:33.273 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:16:33.274 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-30T00:16:33.274 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:16:33.387 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2023-09-30T00:16:33.762 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:33.961 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:34.437 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:16:34.438 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2023-09-30T00:16:34.438 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:16:34.543 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rgw 2023-09-30T00:16:34.740 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:28 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:34.915 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:35.798 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:26 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:36.100 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:36.575 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:16:36.576 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-30T00:16:36.576 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:16:36.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2023-09-30T00:16:37.051 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:37.203 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:37.674 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:16:37.674 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2023-09-30T00:16:37.674 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:16:37.774 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-cephfs 2023-09-30T00:16:37.897 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:31 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:38.146 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:39.003 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:30 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:39.241 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:39.710 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:16:39.711 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-09-30T00:16:39.711 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:16:39.828 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2023-09-30T00:16:40.199 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:40.319 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:40.789 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:16:40.790 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2023-09-30T00:16:40.790 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:16:40.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rbd 2023-09-30T00:16:41.043 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:35 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:41.271 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:42.147 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:33 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:42.930 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:16:42.931 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:42.931 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2023-09-30T00:16:42.931 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:42.931 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:16:42.932 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T00:16:42.932 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:42.932 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:16:42.932 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:42.932 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-30T00:16:42.932 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:42.932 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 90 k 2023-09-30T00:16:42.932 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 264 k 2023-09-30T00:16:42.932 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:16:43.293 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 251 kB/s | 90 kB 00:00 2023-09-30T00:16:43.293 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:16:43.294 INFO:teuthology.orchestra.run.smithi121.stdout:Total 249 kB/s | 90 kB 00:00 2023-09-30T00:16:43.294 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:16:43.305 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:16:43.305 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:16:43.360 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:16:43.365 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:16:43.460 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:43.474 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:16:43.649 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:43.933 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:16:43.934 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2023-09-30T00:16:43.934 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:16:43.947 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:44.044 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-fuse 2023-09-30T00:16:44.413 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:44.966 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:44.966 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:38 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:45.315 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:36 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:16:47.011 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:47.011 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:16:47.011 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:16:47.011 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:47.011 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:16:47.129 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:16:47.130 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:47.130 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2023-09-30T00:16:47.130 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:47.130 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:16:47.130 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T00:16:47.130 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:47.130 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:16:47.130 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:47.130 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2023-09-30T00:16:47.130 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:47.131 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 90 k 2023-09-30T00:16:47.131 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 264 k 2023-09-30T00:16:47.131 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:16:47.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2023-09-30T00:16:47.353 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 406 kB/s | 90 kB 00:00 2023-09-30T00:16:47.354 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:16:47.354 INFO:teuthology.orchestra.run.smithi094.stdout:Total 404 kB/s | 90 kB 00:00 2023-09-30T00:16:47.354 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:16:47.366 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:16:47.366 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:16:47.421 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:16:47.425 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:16:47.517 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:47.544 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:16:47.754 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:48.056 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:48.401 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:42 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:49.151 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:40 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:50.189 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:16:50.190 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2023-09-30T00:16:50.190 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:50.190 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:16:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:16:50.190 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:50.190 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:16:50.190 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:50.190 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-30T00:16:50.191 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:50.191 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.5 M 2023-09-30T00:16:50.191 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 16 M 2023-09-30T00:16:50.191 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:16:50.868 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 5.1 MB/s | 3.5 MB 00:00 2023-09-30T00:16:50.869 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:16:50.869 INFO:teuthology.orchestra.run.smithi121.stdout:Total 5.1 MB/s | 3.5 MB 00:00 2023-09-30T00:16:50.870 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:16:50.883 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:16:50.884 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:16:50.951 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:16:50.955 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:16:51.037 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:16:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:51.135 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:16:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:16:51.135 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:51.135 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:16:51.256 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-mirror 2023-09-30T00:16:51.440 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:16:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:51.629 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:52.457 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:43 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:53.106 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:47 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:54.306 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:16:54.307 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:54.307 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2023-09-30T00:16:54.308 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:54.308 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:16:54.308 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:16:54.308 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:54.308 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:16:54.308 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:16:54.308 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2023-09-30T00:16:54.308 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:54.308 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.5 M 2023-09-30T00:16:54.308 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 16 M 2023-09-30T00:16:54.308 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:16:55.017 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:16:55.115 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:55.115 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:16:55.116 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:16:55.116 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:55.116 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:16:55.255 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 3.6 MB/s | 3.5 MB 00:00 2023-09-30T00:16:55.256 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:16:55.256 INFO:teuthology.orchestra.run.smithi094.stdout:Total 3.6 MB/s | 3.5 MB 00:00 2023-09-30T00:16:55.257 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:16:55.270 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:16:55.270 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:16:55.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2023-09-30T00:16:55.339 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:16:55.343 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:16:55.652 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-09-30T00:16:55.824 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:16:56.029 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:56.471 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:56.487 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:50 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:16:57.480 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:57.480 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:48 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:16:58.342 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-09-30T00:16:58.343 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:58.343 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2023-09-30T00:16:58.343 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:58.343 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-09-30T00:16:58.343 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T00:16:58.343 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:58.343 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-09-30T00:16:58.344 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-09-30T00:16:58.344 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2023-09-30T00:16:58.344 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:16:58.344 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 173 k 2023-09-30T00:16:58.344 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 552 k 2023-09-30T00:16:58.344 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-09-30T00:16:58.602 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 673 kB/s | 173 kB 00:00 2023-09-30T00:16:58.602 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:16:58.602 INFO:teuthology.orchestra.run.smithi121.stdout:Total 670 kB/s | 173 kB 00:00 2023-09-30T00:16:58.602 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-09-30T00:16:58.615 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-09-30T00:16:58.615 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-09-30T00:16:58.670 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-09-30T00:16:58.674 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-09-30T00:16:58.795 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-09-30T00:16:59.003 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:59.275 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:16:59.416 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:16:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:59.513 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:16:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:16:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:16:59.513 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:16:59.631 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd 2023-09-30T00:17:00.008 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-09-30T00:17:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:17:00.264 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:54 ago on Sat 30 Sep 2023 12:13:06 AM UTC. 2023-09-30T00:17:01.079 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:52 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:17:02.307 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-09-30T00:17:02.405 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:02.406 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-09-30T00:17:02.406 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:17:02.406 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:02.406 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-09-30T00:17:02.531 DEBUG:teuthology.parallel:result is None 2023-09-30T00:17:02.943 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-09-30T00:17:02.943 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:17:02.943 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2023-09-30T00:17:02.943 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:17:02.944 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-09-30T00:17:02.944 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T00:17:02.944 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:02.944 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-09-30T00:17:02.944 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-09-30T00:17:02.944 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2023-09-30T00:17:02.944 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:02.945 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 173 k 2023-09-30T00:17:02.945 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 552 k 2023-09-30T00:17:02.945 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-09-30T00:17:03.098 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 1.1 MB/s | 173 kB 00:00 2023-09-30T00:17:03.098 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:17:03.098 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.1 MB/s | 173 kB 00:00 2023-09-30T00:17:03.099 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-09-30T00:17:03.111 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-09-30T00:17:03.111 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-09-30T00:17:03.168 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-09-30T00:17:03.173 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-09-30T00:17:03.304 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-09-30T00:17:03.494 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:17:03.744 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:17:04.760 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:17:04.761 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:55 ago on Sat 30 Sep 2023 12:13:09 AM UTC. 2023-09-30T00:17:06.992 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-09-30T00:17:07.088 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:07.088 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-09-30T00:17:07.088 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:17:07.089 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:07.089 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-09-30T00:17:07.269 DEBUG:teuthology.parallel:result is None 2023-09-30T00:17:07.269 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:17:07.419 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T00:17:07.438 INFO:teuthology.orchestra.run.smithi094.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T00:17:07.438 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T00:17:07.439 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T00:17:07.439 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:17:07.657 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T00:17:07.681 INFO:teuthology.orchestra.run.smithi121.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T00:17:07.681 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T00:17:07.681 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T00:17:07.681 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T00:17:07.682 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:07.682 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:17:07.707 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:07.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:17:07.748 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T00:17:07.749 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:07.749 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:17:07.773 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:17:07.838 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:07.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:17:07.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:17:07.930 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T00:17:07.930 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:07.931 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:17:07.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:17:08.018 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:08.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:17:08.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:17:08.108 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T00:17:08.108 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:08.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:17:08.132 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:17:08.196 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:08.196 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:17:08.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:17:08.284 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T00:17:08.344 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T00:17:08.345 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /var/log/ceph 2023-09-30T00:17:08.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2023-09-30T00:17:08.370 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T00:17:08.371 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:17:08.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:17:08.434 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T00:17:08.435 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:17:08.475 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:17:08.507 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-09-30T00:17:08.507 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-09-30T00:17:08.508 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-09-30T00:17:08.508 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:08.508 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-09-30T00:17:08.551 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-09-30T00:17:08.663 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-09-30T00:17:08.688 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:08.688 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-09-30T00:17:08.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-09-30T00:17:08.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-09-30T00:17:08.842 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T00:17:08.842 INFO:tasks.ceph:config {'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 3}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mgr ttl cache expire seconds': 5}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_', 'replacing it with standby', 'No standby daemons available', 'Reduced data availability', 'Degraded data redundancy', 'objects misplaced', 'Synthetic exception in serve', 'influxdb python module not found', '\\(MGR_ZABBIX_', 'foo bar', 'Failed to open Telegraf', 'evicting unresponsive client', '1 mgr modules have recently crashed \\(RECENT_MGR_MODULE_CRASH\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T00:17:08.843 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{rhel_8} tasks/module_selftest}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406690', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 2, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 3}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mgr ttl cache expire seconds': 5}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'], ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi094.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMqbQIXsPQL+Ar4JA2DrEuvSb1C56pvU9IeQTqdHEFydStZM3Jh68Ce792vRf9g1fPDGyeeDxExrc5ud827ofuI=', 'smithi121.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG/8gFow/QHseLGfYRci2qn39BiLQNOI1wirw6FMtOPWoqfPf2NxV7uScS9oAVi0/HZb9BM2iGwESeViMP3MP7c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_', 'replacing it with standby', 'No standby daemons available', 'Reduced data availability', 'Degraded data redundancy', 'objects misplaced', 'Synthetic exception in serve', 'influxdb python module not found', '\\(MGR_ZABBIX_', 'foo bar', 'Failed to open Telegraf', 'evicting unresponsive client', '1 mgr modules have recently crashed \\(RECENT_MGR_MODULE_CRASH\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'wait-for-scrub': False, 'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 3}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mgr ttl cache expire seconds': 5}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'cluster': 'ceph'}}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.mgr.test_module_selftest']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T00:17:08.843 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:17:08.845 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:17:08.893 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:17:08.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:17:08.957 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:08.957 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:17:08.978 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:17:08.978 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2023-09-30T00:17:09.033 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:17:09.033 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:17:09.033 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 219765 Links: 1 2023-09-30T00:17:09.033 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:17:09.033 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:17:09.034 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2023-09-30 00:16:56.242464711 +0000 2023-09-30T00:17:09.034 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2023-09-30 00:12:22.035062720 +0000 2023-09-30T00:17:09.034 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2023-09-30 00:12:22.035062720 +0000 2023-09-30T00:17:09.034 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2023-09-30T00:17:09.034 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:17:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2023-09-30T00:17:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2023-09-30T00:17:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000124858 s, 4.1 MB/s 2023-09-30T00:17:09.097 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:17:09.153 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2023-09-30T00:17:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:17:09.208 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:17:09.208 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 216546 Links: 1 2023-09-30T00:17:09.208 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:17:09.208 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:17:09.209 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2023-09-30 00:16:56.242464711 +0000 2023-09-30T00:17:09.209 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2023-09-30 00:12:22.535054477 +0000 2023-09-30T00:17:09.209 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2023-09-30 00:12:22.535054477 +0000 2023-09-30T00:17:09.209 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2023-09-30T00:17:09.209 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:17:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2023-09-30T00:17:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2023-09-30T00:17:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000111545 s, 4.6 MB/s 2023-09-30T00:17:09.272 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:17:09.328 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2023-09-30T00:17:09.383 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:17:09.383 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:17:09.383 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 219136 Links: 1 2023-09-30T00:17:09.383 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:17:09.383 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:17:09.383 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2023-09-30 00:16:56.242464711 +0000 2023-09-30T00:17:09.383 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2023-09-30 00:12:23.036046217 +0000 2023-09-30T00:17:09.383 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2023-09-30 00:12:23.036046217 +0000 2023-09-30T00:17:09.383 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2023-09-30T00:17:09.384 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:17:09.445 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2023-09-30T00:17:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2023-09-30T00:17:09.446 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000114862 s, 4.5 MB/s 2023-09-30T00:17:09.447 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:17:09.502 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2023-09-30T00:17:09.557 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:17:09.557 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:17:09.557 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 223310 Links: 1 2023-09-30T00:17:09.557 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:17:09.557 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:17:09.557 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2023-09-30 00:16:56.242464711 +0000 2023-09-30T00:17:09.557 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2023-09-30 00:12:23.520038237 +0000 2023-09-30T00:17:09.557 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2023-09-30 00:12:23.520038237 +0000 2023-09-30T00:17:09.558 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2023-09-30T00:17:09.558 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:17:09.620 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2023-09-30T00:17:09.620 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2023-09-30T00:17:09.620 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000111897 s, 4.6 MB/s 2023-09-30T00:17:09.621 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:17:09.677 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1'} 2023-09-30T00:17:09.677 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:09.677 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:17:09.692 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:17:09.693 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2023-09-30T00:17:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:17:09.749 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:17:09.749 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 221361 Links: 1 2023-09-30T00:17:09.749 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:17:09.749 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:17:09.749 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-09-30 00:16:51.821120050 +0000 2023-09-30T00:17:09.749 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-09-30 00:12:23.676466015 +0000 2023-09-30T00:17:09.750 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-09-30 00:12:23.676466015 +0000 2023-09-30T00:17:09.750 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-09-30T00:17:09.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:17:09.813 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-09-30T00:17:09.814 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-09-30T00:17:09.814 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000234502 s, 2.2 MB/s 2023-09-30T00:17:09.815 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:17:09.871 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2023-09-30T00:17:09.928 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:17:09.928 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:17:09.928 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 216825 Links: 1 2023-09-30T00:17:09.928 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:17:09.928 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:17:09.928 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-09-30 00:16:51.821120050 +0000 2023-09-30T00:17:09.928 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-09-30 00:12:24.184459550 +0000 2023-09-30T00:17:09.928 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-09-30 00:12:24.184459550 +0000 2023-09-30T00:17:09.928 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-09-30T00:17:09.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:17:09.992 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-09-30T00:17:09.992 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-09-30T00:17:09.992 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000263386 s, 1.9 MB/s 2023-09-30T00:17:09.993 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:17:10.050 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2023-09-30T00:17:10.105 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:17:10.105 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:17:10.105 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 218755 Links: 1 2023-09-30T00:17:10.105 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:17:10.105 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:17:10.105 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-09-30 00:16:51.821120050 +0000 2023-09-30T00:17:10.105 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-09-30 00:12:24.685453175 +0000 2023-09-30T00:17:10.106 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-09-30 00:12:24.685453175 +0000 2023-09-30T00:17:10.106 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-09-30T00:17:10.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:17:10.168 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-09-30T00:17:10.168 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-09-30T00:17:10.168 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000195908 s, 2.6 MB/s 2023-09-30T00:17:10.169 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:17:10.225 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2023-09-30T00:17:10.281 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:17:10.281 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:17:10.281 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 218850 Links: 1 2023-09-30T00:17:10.281 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:17:10.281 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:17:10.281 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-09-30 00:16:51.821120050 +0000 2023-09-30T00:17:10.281 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-09-30 00:12:25.185446812 +0000 2023-09-30T00:17:10.282 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-09-30 00:12:25.185446812 +0000 2023-09-30T00:17:10.282 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-09-30T00:17:10.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:17:10.343 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-09-30T00:17:10.344 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-09-30T00:17:10.344 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000232134 s, 2.2 MB/s 2023-09-30T00:17:10.345 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:17:10.400 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2023-09-30T00:17:10.401 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2023-09-30T00:17:10.401 INFO:tasks.ceph:Generating config... 2023-09-30T00:17:10.403 INFO:tasks.ceph:[client] debug client = 20 2023-09-30T00:17:10.403 INFO:tasks.ceph:[client] debug mgrc = 20 2023-09-30T00:17:10.403 INFO:tasks.ceph:[client] debug ms = 1 2023-09-30T00:17:10.404 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-09-30T00:17:10.404 INFO:tasks.ceph:[mds] debug mgrc = 20 2023-09-30T00:17:10.404 INFO:tasks.ceph:[mgr] debug client = 20 2023-09-30T00:17:10.404 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T00:17:10.404 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T00:17:10.404 INFO:tasks.ceph:[mgr] mgr ttl cache expire seconds = 5 2023-09-30T00:17:10.405 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T00:17:10.405 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T00:17:10.405 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T00:17:10.405 INFO:tasks.ceph:[osd] bdev async discard = True 2023-09-30T00:17:10.405 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-09-30T00:17:10.405 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-09-30T00:17:10.406 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T00:17:10.406 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T00:17:10.406 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-30T00:17:10.406 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-30T00:17:10.406 INFO:tasks.ceph:[osd] debug mgrc = 20 2023-09-30T00:17:10.406 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T00:17:10.407 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T00:17:10.407 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T00:17:10.407 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T00:17:10.407 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T00:17:10.407 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T00:17:10.407 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T00:17:10.407 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T00:17:10.408 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T00:17:10.408 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T00:17:10.458 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T00:17:10.461 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T00:17:10.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:17:10.564 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.94'), ('mon.c', '[v2:172.21.15.94:3301,v1:172.21.15.94:6790]'), ('mon.b', '172.21.15.121')] 2023-09-30T00:17:10.565 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.94,[v2:172.21.15.94:3301,v1:172.21.15.94:6790],172.21.15.121', 'mon election default strategy': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'mgr ttl cache expire seconds': 5}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds': {'debug mgrc': 20}} 2023-09-30T00:17:10.566 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:10.566 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:17:10.620 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.94 --addv c '[v2:172.21.15.94:3301,v1:172.21.15.94:6790]' --add b 172.21.15.121 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:17:10.702 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:17:10.702 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: generated fsid 3c891173-587e-4c6f-9c6a-04ec750eedad 2023-09-30T00:17:10.702 INFO:teuthology.orchestra.run.smithi094.stdout:setting min_mon_release = octopus 2023-09-30T00:17:10.703 INFO:teuthology.orchestra.run.smithi094.stdout:epoch 0 2023-09-30T00:17:10.703 INFO:teuthology.orchestra.run.smithi094.stdout:fsid 3c891173-587e-4c6f-9c6a-04ec750eedad 2023-09-30T00:17:10.703 INFO:teuthology.orchestra.run.smithi094.stdout:last_changed 2023-09-30T00:17:10.701144+0000 2023-09-30T00:17:10.703 INFO:teuthology.orchestra.run.smithi094.stdout:created 2023-09-30T00:17:10.701144+0000 2023-09-30T00:17:10.703 INFO:teuthology.orchestra.run.smithi094.stdout:min_mon_release 15 (octopus) 2023-09-30T00:17:10.704 INFO:teuthology.orchestra.run.smithi094.stdout:election_strategy: 3 2023-09-30T00:17:10.704 INFO:teuthology.orchestra.run.smithi094.stdout:0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2023-09-30T00:17:10.704 INFO:teuthology.orchestra.run.smithi094.stdout:1: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.b 2023-09-30T00:17:10.704 INFO:teuthology.orchestra.run.smithi094.stdout:2: [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] mon.c 2023-09-30T00:17:10.704 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-30T00:17:10.705 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:17:10.757 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3c891173-587e-4c6f-9c6a-04ec750eedad... 2023-09-30T00:17:10.758 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:17:10.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:17:10.839 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2023-09-30T00:17:10.839 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2023-09-30T00:17:10.839 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:17:10.840 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2023-09-30T00:17:10.840 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.840 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2023-09-30T00:17:10.840 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.840 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2023-09-30T00:17:10.840 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.841 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2023-09-30T00:17:10.841 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2023-09-30T00:17:10.841 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.841 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2023-09-30T00:17:10.841 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2023-09-30T00:17:10.841 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.842 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:17:10.842 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2023-09-30T00:17:10.842 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.842 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:17:10.842 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:17:10.843 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.843 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2023-09-30T00:17:10.843 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.843 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2023-09-30T00:17:10.843 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2023-09-30T00:17:10.843 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:17:10.844 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:17:10.844 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2023-09-30T00:17:10.844 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:17:10.844 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2023-09-30T00:17:10.844 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:17:10.844 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:17:10.845 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2023-09-30T00:17:10.845 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.845 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:17:10.845 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.845 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2023-09-30T00:17:10.845 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.845 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2023-09-30T00:17:10.845 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.846 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2023-09-30T00:17:10.846 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2023-09-30T00:17:10.846 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2023-09-30T00:17:10.846 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = "172.21.15.94,[v2:172.21.15.94:3301,v1:172.21.15.94:6790],172.21.15.121" 2023-09-30T00:17:10.846 INFO:teuthology.orchestra.run.smithi094.stdout: mon election default strategy = 3 2023-09-30T00:17:10.846 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = 3c891173-587e-4c6f-9c6a-04ec750eedad 2023-09-30T00:17:10.846 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.846 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2023-09-30T00:17:10.846 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2023-09-30T00:17:10.846 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.846 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2023-09-30T00:17:10.846 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 600 2023-09-30T00:17:10.847 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:17:10.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.847 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2023-09-30T00:17:10.847 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2023-09-30T00:17:10.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.847 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2023-09-30T00:17:10.847 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2023-09-30T00:17:10.847 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2023-09-30T00:17:10.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.847 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2023-09-30T00:17:10.847 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2023-09-30T00:17:10.848 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.848 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:17:10.848 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.848 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2023-09-30T00:17:10.848 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.848 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2023-09-30T00:17:10.848 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.848 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:17:10.848 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2023-09-30T00:17:10.848 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.848 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2023-09-30T00:17:10.848 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2023-09-30T00:17:10.849 INFO:teuthology.orchestra.run.smithi094.stdout: bdev async discard = True 2023-09-30T00:17:10.849 INFO:teuthology.orchestra.run.smithi094.stdout: bdev enable discard = True 2023-09-30T00:17:10.849 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore allocator = bitmap 2023-09-30T00:17:10.849 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore block size = 96636764160 2023-09-30T00:17:10.849 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore fsck on mount = True 2023-09-30T00:17:10.849 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluefs = 20 2023-09-30T00:17:10.849 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluestore = 20 2023-09-30T00:17:10.849 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgrc = 20 2023-09-30T00:17:10.849 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2023-09-30T00:17:10.849 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 20 2023-09-30T00:17:10.849 INFO:teuthology.orchestra.run.smithi094.stdout: debug rocksdb = 10 2023-09-30T00:17:10.850 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T00:17:10.850 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd full ratio = 0.9 2023-09-30T00:17:10.850 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd nearfull ratio = 0.8 2023-09-30T00:17:10.850 INFO:teuthology.orchestra.run.smithi094.stdout: osd failsafe full ratio = 0.95 2023-09-30T00:17:10.850 INFO:teuthology.orchestra.run.smithi094.stdout: osd objectstore = bluestore 2023-09-30T00:17:10.850 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.850 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2023-09-30T00:17:10.850 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2023-09-30T00:17:10.850 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2023-09-30T00:17:10.850 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2023-09-30T00:17:10.850 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2023-09-30T00:17:10.851 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:17:10.851 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:17:10.851 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2023-09-30T00:17:10.851 INFO:teuthology.orchestra.run.smithi094.stdout: debug client = 20 2023-09-30T00:17:10.851 INFO:teuthology.orchestra.run.smithi094.stdout: mgr ttl cache expire seconds = 5 2023-09-30T00:17:10.851 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.851 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2023-09-30T00:17:10.851 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2023-09-30T00:17:10.851 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2023-09-30T00:17:10.851 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2023-09-30T00:17:10.851 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2023-09-30T00:17:10.851 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2023-09-30T00:17:10.852 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2023-09-30T00:17:10.852 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:17:10.852 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2023-09-30T00:17:10.852 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2023-09-30T00:17:10.852 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:17:10.852 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.852 INFO:teuthology.orchestra.run.smithi094.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:17:10.852 INFO:teuthology.orchestra.run.smithi094.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:17:10.852 INFO:teuthology.orchestra.run.smithi094.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:17:10.852 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.852 INFO:teuthology.orchestra.run.smithi094.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:17:10.852 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:17:10.853 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:17:10.853 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.853 INFO:teuthology.orchestra.run.smithi094.stdout: # 1m isn't quite enough 2023-09-30T00:17:10.853 INFO:teuthology.orchestra.run.smithi094.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:17:10.853 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.853 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:17:10.853 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:10.853 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2023-09-30T00:17:10.853 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2023-09-30T00:17:10.853 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2023-09-30T00:17:10.853 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2023-09-30T00:17:10.853 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:17:10.854 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:17:10.854 INFO:teuthology.orchestra.run.smithi094.stdout: debug client = 20 2023-09-30T00:17:10.854 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgrc = 20 2023-09-30T00:17:10.854 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2023-09-30T00:17:10.854 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2023-09-30T00:17:10.854 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.c] 2023-09-30T00:17:10.854 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.b] 2023-09-30T00:17:10.854 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.a] 2023-09-30T00:17:10.854 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.c] 2023-09-30T00:17:10.855 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.b] 2023-09-30T00:17:10.855 INFO:teuthology.orchestra.run.smithi094.stdout:[mds] 2023-09-30T00:17:10.855 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgrc = 20 2023-09-30T00:17:10.856 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2023-09-30T00:17:10.856 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2023-09-30T00:17:10.856 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:17:10.856 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2023-09-30T00:17:10.857 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.857 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2023-09-30T00:17:10.857 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.857 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2023-09-30T00:17:10.857 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.857 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2023-09-30T00:17:10.857 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2023-09-30T00:17:10.857 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.857 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2023-09-30T00:17:10.857 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2023-09-30T00:17:10.857 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.858 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:17:10.858 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2023-09-30T00:17:10.858 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.858 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:17:10.858 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:17:10.858 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.858 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2023-09-30T00:17:10.858 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.858 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2023-09-30T00:17:10.858 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2023-09-30T00:17:10.858 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:17:10.859 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:17:10.859 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2023-09-30T00:17:10.859 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:17:10.859 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2023-09-30T00:17:10.859 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:17:10.859 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:17:10.859 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2023-09-30T00:17:10.859 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.859 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:17:10.859 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.859 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2023-09-30T00:17:10.859 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.860 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2023-09-30T00:17:10.860 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.860 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2023-09-30T00:17:10.860 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2023-09-30T00:17:10.860 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2023-09-30T00:17:10.860 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.94,[v2:172.21.15.94:3301,v1:172.21.15.94:6790],172.21.15.121" 2023-09-30T00:17:10.860 INFO:teuthology.orchestra.run.smithi121.stdout: mon election default strategy = 3 2023-09-30T00:17:10.860 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 3c891173-587e-4c6f-9c6a-04ec750eedad 2023-09-30T00:17:10.860 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.860 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2023-09-30T00:17:10.860 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2023-09-30T00:17:10.861 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.861 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2023-09-30T00:17:10.861 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2023-09-30T00:17:10.861 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:17:10.861 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.861 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2023-09-30T00:17:10.861 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2023-09-30T00:17:10.861 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.861 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2023-09-30T00:17:10.861 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2023-09-30T00:17:10.861 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2023-09-30T00:17:10.861 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.862 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2023-09-30T00:17:10.862 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2023-09-30T00:17:10.862 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.862 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:17:10.862 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.862 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2023-09-30T00:17:10.862 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.862 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2023-09-30T00:17:10.862 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.862 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:17:10.862 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2023-09-30T00:17:10.863 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.863 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2023-09-30T00:17:10.863 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2023-09-30T00:17:10.863 INFO:teuthology.orchestra.run.smithi121.stdout: bdev async discard = True 2023-09-30T00:17:10.863 INFO:teuthology.orchestra.run.smithi121.stdout: bdev enable discard = True 2023-09-30T00:17:10.863 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore allocator = bitmap 2023-09-30T00:17:10.863 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2023-09-30T00:17:10.863 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2023-09-30T00:17:10.863 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 20 2023-09-30T00:17:10.863 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 20 2023-09-30T00:17:10.863 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgrc = 20 2023-09-30T00:17:10.864 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-09-30T00:17:10.864 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2023-09-30T00:17:10.864 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 10 2023-09-30T00:17:10.864 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T00:17:10.864 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2023-09-30T00:17:10.864 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2023-09-30T00:17:10.864 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2023-09-30T00:17:10.864 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2023-09-30T00:17:10.864 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.864 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2023-09-30T00:17:10.864 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-09-30T00:17:10.864 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2023-09-30T00:17:10.865 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2023-09-30T00:17:10.865 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2023-09-30T00:17:10.865 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:17:10.865 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:17:10.865 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2023-09-30T00:17:10.865 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2023-09-30T00:17:10.865 INFO:teuthology.orchestra.run.smithi121.stdout: mgr ttl cache expire seconds = 5 2023-09-30T00:17:10.865 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.865 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2023-09-30T00:17:10.865 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-09-30T00:17:10.865 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2023-09-30T00:17:10.866 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2023-09-30T00:17:10.866 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2023-09-30T00:17:10.866 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2023-09-30T00:17:10.866 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2023-09-30T00:17:10.866 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:17:10.866 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2023-09-30T00:17:10.866 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2023-09-30T00:17:10.866 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:17:10.866 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.866 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:17:10.866 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:17:10.866 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:17:10.867 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.867 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:17:10.867 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:17:10.867 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:17:10.867 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.867 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2023-09-30T00:17:10.867 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:17:10.867 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.867 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:17:10.867 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-09-30T00:17:10.867 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2023-09-30T00:17:10.868 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2023-09-30T00:17:10.868 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2023-09-30T00:17:10.868 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2023-09-30T00:17:10.868 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:17:10.868 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:17:10.868 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2023-09-30T00:17:10.868 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgrc = 20 2023-09-30T00:17:10.868 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-09-30T00:17:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2023-09-30T00:17:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2023-09-30T00:17:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2023-09-30T00:17:10.868 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2023-09-30T00:17:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2023-09-30T00:17:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2023-09-30T00:17:10.869 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2023-09-30T00:17:10.869 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgrc = 20 2023-09-30T00:17:10.869 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T00:17:10.869 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T00:17:10.946 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T00:17:10.947 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:10.947 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T00:17:10.961 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:10.961 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T00:17:11.015 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2023-09-30T00:17:11.015 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:11.015 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:17:11.016 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:17:11.090 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:11.090 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:17:11.145 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2023-09-30T00:17:11.145 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:11.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:17:11.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:17:11.179 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:11.179 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:17:11.233 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T00:17:11.234 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T00:17:11.234 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:17:11.281 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:17:11.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T00:17:11.353 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T00:17:11.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-z && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.z /var/lib/ceph/mgr/ceph-z/keyring 2023-09-30T00:17:11.403 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-z/keyring 2023-09-30T00:17:11.405 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T00:17:11.405 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-09-30T00:17:11.453 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-09-30T00:17:11.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-09-30T00:17:11.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-09-30T00:17:11.566 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-09-30T00:17:11.568 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-09-30T00:17:11.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-09-30T00:17:11.640 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-09-30T00:17:11.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-09-30T00:17:11.667 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T00:17:11.668 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T00:17:11.706 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T00:17:11.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-09-30T00:17:11.755 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-09-30T00:17:11.768 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T00:17:11.768 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2023-09-30T00:17:11.768 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T00:17:11.792 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1'} 2023-09-30T00:17:11.792 INFO:tasks.ceph:role: osd.0 2023-09-30T00:17:11.792 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com 2023-09-30T00:17:11.793 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:17:11.856 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:17:11.856 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:17:11.856 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:17:11.856 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 2023-09-30T00:17:11.857 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:17:11.857 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:17:11.857 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:17:11.857 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:17:11.857 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:17:11.857 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:17:11.879 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2023-09-30T00:17:11.880 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2023-09-30T00:17:11.880 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T00:17:11.958 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T00:17:12.027 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:17:12.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:12.132+0000 7f5d126cb540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T00:17:12.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:12.132+0000 7f5d126cb540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T00:17:12.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:12.132+0000 7f5d126cb540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:17:12.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:12.132+0000 7f5d126cb540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:17:12.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:12.132+0000 7f5d126cb540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:17:12.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:12.132+0000 7f5d126cb540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T00:17:12.654 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:12.653+0000 7f5d126cb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:17:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:13.416+0000 7f5d126cb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:17:14.178 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:14.178+0000 7f5d126cb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:17:14.971 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T00:17:15.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T00:17:15.026 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2023-09-30T00:17:15.026 INFO:tasks.ceph:role: osd.1 2023-09-30T00:17:15.027 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2023-09-30T00:17:15.027 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:17:15.100 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:17:15.100 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:17:15.101 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:17:15.101 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 2023-09-30T00:17:15.101 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:17:15.101 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:17:15.101 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:17:15.101 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:17:15.101 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:17:15.101 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:17:15.528 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-09-30T00:17:15.530 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-09-30T00:17:15.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2023-09-30T00:17:15.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T00:17:15.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T00:17:15.799 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2023-09-30T00:17:15.799 INFO:tasks.ceph:role: osd.2 2023-09-30T00:17:15.799 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2023-09-30T00:17:15.799 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:17:15.871 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:17:15.871 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:17:15.871 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:17:15.871 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 2023-09-30T00:17:15.872 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:17:15.872 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:17:15.872 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:17:15.872 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:17:15.872 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:17:15.872 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:17:16.301 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-09-30T00:17:16.303 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-09-30T00:17:16.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2023-09-30T00:17:16.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T00:17:16.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T00:17:16.504 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2023-09-30T00:17:16.505 INFO:tasks.ceph:role: osd.3 2023-09-30T00:17:16.505 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2023-09-30T00:17:16.505 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:17:16.577 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:17:16.577 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:17:16.577 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:17:16.578 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 2023-09-30T00:17:16.578 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:17:16.578 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:17:16.578 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:17:16.578 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:17:16.578 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:17:16.578 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:17:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-09-30T00:17:17.028 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-09-30T00:17:17.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2023-09-30T00:17:17.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T00:17:17.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:17:17.298 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:17.297+0000 7f2b24e08540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T00:17:17.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:17.299+0000 7f2b24e08540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T00:17:17.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:17.299+0000 7f2b24e08540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:17:17.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:17.299+0000 7f2b24e08540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:17:17.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:17.299+0000 7f2b24e08540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:17:17.307 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:17.306+0000 7f2b24e08540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T00:17:17.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:17.834+0000 7f2b24e08540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:17:18.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:18.594+0000 7f2b24e08540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:17:19.359 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:19.358+0000 7f2b24e08540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:17:20.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T00:17:20.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:17:20.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:20.264+0000 7f7137a84540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T00:17:20.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:20.264+0000 7f7137a84540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T00:17:20.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:20.264+0000 7f7137a84540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:17:20.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:20.264+0000 7f7137a84540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:17:20.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:20.264+0000 7f7137a84540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:17:20.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:20.265+0000 7f7137a84540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T00:17:20.790 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:20.789+0000 7f7137a84540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:17:21.552 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:21.551+0000 7f7137a84540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:17:22.314 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:22.313+0000 7f7137a84540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:17:23.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T00:17:23.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:17:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:23.226+0000 7ffa38035540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T00:17:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:23.227+0000 7ffa38035540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T00:17:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:23.227+0000 7ffa38035540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:17:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:23.227+0000 7ffa38035540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:17:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:23.227+0000 7ffa38035540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:17:23.228 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:23.227+0000 7ffa38035540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-30T00:17:23.755 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:23.754+0000 7ffa38035540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:17:24.518 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:24.517+0000 7ffa38035540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:17:25.279 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:25.278+0000 7ffa38035540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:17:26.080 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T00:17:26.148 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T00:17:26.148 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:26.148 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T00:17:26.177 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:26.177 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-09-30T00:17:26.244 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:26.244 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-09-30T00:17:26.311 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:26.311 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T00:17:26.377 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:26.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-09-30T00:17:26.406 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:26.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-z/keyring of=/dev/stdout 2023-09-30T00:17:26.473 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:26.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-09-30T00:17:26.539 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:26.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T00:17:26.606 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:26.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T00:17:26.672 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:26.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T00:17:26.738 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:26.738 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T00:17:26.755 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:26.755 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-09-30T00:17:26.793 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T00:17:26.793 DEBUG:teuthology.orchestra.run.smithi094:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:17:26.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:17:26.835 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.x] 2023-09-30T00:17:26.836 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAHaRdlHz7NEBAAl6KdlvxPYlFqvFoFe72o0g== 2023-09-30T00:17:26.836 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.a] 2023-09-30T00:17:26.836 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAHaRdl81IHGxAAWrc8YVyHUEiwK/+8bfa2Tw== 2023-09-30T00:17:26.836 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.c] 2023-09-30T00:17:26.836 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAHaRdl8UfNIRAAQzd2/3aBZkIfafrsqEU9ag== 2023-09-30T00:17:26.836 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2023-09-30T00:17:26.836 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAIaRdlnlruBxAAjAhwt756T9D1szFZP3YS6Q== 2023-09-30T00:17:26.836 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.y] 2023-09-30T00:17:26.837 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAHaRdljuYYFRAAz5oaTiStmf4a9xL4nuDrjg== 2023-09-30T00:17:26.837 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.z] 2023-09-30T00:17:26.837 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAHaRdl93MPGBAAKyvtzgosMlV52+xDGpzSfA== 2023-09-30T00:17:26.837 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.b] 2023-09-30T00:17:26.837 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAHaRdlMqg5JhAArbQJf4OZUV1KQq2t0tpe6A== 2023-09-30T00:17:26.837 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2023-09-30T00:17:26.837 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQANaRdlZ4DPERAAeY+4EVmmUYAqpAFAzare8A== 2023-09-30T00:17:26.837 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2023-09-30T00:17:26.837 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAQaRdlqTHVDxAAKU5J9ht8DnBimmTbhgvuTQ== 2023-09-30T00:17:26.837 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.3] 2023-09-30T00:17:26.837 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQATaRdlmwOaDRAAz//x24wY1eXNkQXizXmeUg== 2023-09-30T00:17:26.838 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2023-09-30T00:17:26.838 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAHaRdl338PKhAAyegTWhl+zzpj48DTF08q7g== 2023-09-30T00:17:26.838 INFO:teuthology.orchestra.run.smithi094.stdout:[client.1] 2023-09-30T00:17:26.838 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAHaRdl/ckOLRAAI8uD05gvfyCI9xIzmebEhw== 2023-09-30T00:17:26.857 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2023-09-30T00:17:26.857 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAHaRdlHz7NEBAAl6KdlvxPYlFqvFoFe72o0g== 2023-09-30T00:17:26.857 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2023-09-30T00:17:26.857 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAHaRdl81IHGxAAWrc8YVyHUEiwK/+8bfa2Tw== 2023-09-30T00:17:26.857 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2023-09-30T00:17:26.857 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAHaRdl8UfNIRAAQzd2/3aBZkIfafrsqEU9ag== 2023-09-30T00:17:26.857 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2023-09-30T00:17:26.857 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAIaRdlnlruBxAAjAhwt756T9D1szFZP3YS6Q== 2023-09-30T00:17:26.857 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.y] 2023-09-30T00:17:26.857 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAHaRdljuYYFRAAz5oaTiStmf4a9xL4nuDrjg== 2023-09-30T00:17:26.858 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.z] 2023-09-30T00:17:26.858 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAHaRdl93MPGBAAKyvtzgosMlV52+xDGpzSfA== 2023-09-30T00:17:26.858 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2023-09-30T00:17:26.858 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAHaRdlMqg5JhAArbQJf4OZUV1KQq2t0tpe6A== 2023-09-30T00:17:26.858 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2023-09-30T00:17:26.858 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQANaRdlZ4DPERAAeY+4EVmmUYAqpAFAzare8A== 2023-09-30T00:17:26.858 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2023-09-30T00:17:26.858 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAQaRdlqTHVDxAAKU5J9ht8DnBimmTbhgvuTQ== 2023-09-30T00:17:26.858 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2023-09-30T00:17:26.858 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQATaRdlmwOaDRAAz//x24wY1eXNkQXizXmeUg== 2023-09-30T00:17:26.859 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2023-09-30T00:17:26.859 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAHaRdl338PKhAAyegTWhl+zzpj48DTF08q7g== 2023-09-30T00:17:26.859 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2023-09-30T00:17:26.859 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAHaRdl/ckOLRAAI8uD05gvfyCI9xIzmebEhw== 2023-09-30T00:17:26.860 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:17:26.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:17:26.948 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-09-30T00:17:26.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-09-30T00:17:27.001 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-09-30T00:17:27.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-09-30T00:17:27.052 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:17:27.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:17:27.100 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:17:27.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:17:27.149 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.z --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:17:27.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.z --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:17:27.198 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-09-30T00:17:27.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-09-30T00:17:27.250 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:17:27.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:17:27.299 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:17:27.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:17:27.347 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:17:27.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:17:27.396 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:17:27.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:17:27.448 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:17:27.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:17:27.500 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T00:17:27.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T00:17:27.524 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:17:27.615 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T00:17:27.639 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T00:17:27.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:17:27.794 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T00:17:27.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T00:17:27.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:17:27.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T00:17:27.961 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:17:27.963 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:17:28.015 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T00:17:28.015 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T00:17:28.016 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-30T00:17:28.018 INFO:tasks.ceph.mon.a:Started 2023-09-30T00:17:28.018 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T00:17:28.018 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-30T00:17:28.020 INFO:tasks.ceph.mon.c:Started 2023-09-30T00:17:28.020 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T00:17:28.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-30T00:17:28.058 INFO:tasks.ceph.mon.b:Started 2023-09-30T00:17:28.058 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T00:17:28.059 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T00:17:28.059 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-30T00:17:28.061 INFO:tasks.ceph.mgr.x:Started 2023-09-30T00:17:28.061 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-09-30T00:17:28.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-09-30T00:17:28.063 INFO:tasks.ceph.mgr.y:Started 2023-09-30T00:17:28.063 INFO:tasks.ceph.mgr.z:Restarting daemon 2023-09-30T00:17:28.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i z 2023-09-30T00:17:28.065 INFO:tasks.ceph.mgr.z:Started 2023-09-30T00:17:28.065 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T00:17:28.065 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T00:17:28.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.182+0000 7f7401908700 1 Processor -- start 2023-09-30T00:17:28.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.182+0000 7f7401908700 1 -- start start 2023-09-30T00:17:28.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.182+0000 7f7401908700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:28.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.182+0000 7f7401908700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:28.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.182+0000 7f7401908700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc06e4a0 con 0x7f73fc06a880 2023-09-30T00:17:28.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.182+0000 7f7401908700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73fc06ae10 con 0x7f73fc06e0d0 2023-09-30T00:17:28.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.182+0000 7f7401908700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0a60d0 con 0x7f73fc0a6900 2023-09-30T00:17:28.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.182+0000 7f7400906700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:28.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.182+0000 7f7400906700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41142/0 (socket says 172.21.15.94:41142) 2023-09-30T00:17:28.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.182+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 learned_addr learned my addr 172.21.15.94:0/3956885928 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:28.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.182+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:28.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.183+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 msgr2=0x7f73fc0a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:17:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.183+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 msgr2=0x7f73fc0a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.183+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.183+0000 7f73fbfff700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73fc06a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:17:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.183+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:17:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.183+0000 7f73fbfff700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73fc06a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.183+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.183+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:17:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.384+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.384+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:28.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.384+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 msgr2=0x7f73fc0a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:17:28.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.384+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 msgr2=0x7f73fc0a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:28.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.384+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:28.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.384+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:17:28.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.384+0000 7f73fbfff700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73fc06a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:17:28.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.384+0000 7f73fbfff700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73fc06a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:28.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.384+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:28.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.384+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:17:28.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.785+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:28.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.785+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:28.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.785+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 msgr2=0x7f73fc0a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:17:28.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.785+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 msgr2=0x7f73fc0a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:28.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.785+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:28.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.785+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:17:28.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.785+0000 7f73fbfff700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73fc06a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:17:28.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.785+0000 7f73fbfff700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73fc06a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:28.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.785+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:28.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:28.785+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:17:29.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:29.586+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:29.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:29.586+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:29.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:29.586+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 msgr2=0x7f73fc0a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:17:29.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:29.586+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 msgr2=0x7f73fc0a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:29.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:29.586+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:29.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:29.586+0000 7f73fbfff700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73fc06a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:17:29.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:29.586+0000 7f73fbfff700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73fc06a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:29.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:29.586+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:29.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:29.586+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:17:29.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:29.586+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:17:31.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f73fa7fc700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73fc06a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:17:31.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f73fa7fc700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73fc06a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:31.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f73fa7fc700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 msgr2=0x7f73fc0a56a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:17:31.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f73fa7fc700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73fc0a56a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:31.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f73fa7fc700 1 -- 172.21.15.94:0/3956885928 >> v1:172.21.15.121:6789/0 conn(0x7f73fc06a880 legacy=0x7f73fc140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:31.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f73fa7fc700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:31.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f73fa7fc700 1 -- 172.21.15.94:0/3956885928 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc140950 con 0x7f73e80061b0 2023-09-30T00:17:31.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f73fa7fc700 1 -- 172.21.15.94:0/3956885928 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73fc06ae10 con 0x7f73e8003730 2023-09-30T00:17:31.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:31.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 msgr2=0x7f73e8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:17:31.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 msgr2=0x7f73e8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:31.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:31.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.183+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:17:31.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.384+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:31.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.384+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 msgr2=0x7f73e8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:17:31.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.384+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 msgr2=0x7f73e8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:31.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.384+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:31.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.384+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:17:31.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.785+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:31.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.785+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 msgr2=0x7f73e8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:17:31.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.785+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 msgr2=0x7f73e8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:31.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.785+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:31.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:31.785+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:17:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:32.586+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:32.586+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 msgr2=0x7f73e8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:17:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:32.586+0000 7f7400906700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 msgr2=0x7f73e8005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:32.586+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:32.586+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:17:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fa7fc700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 msgr2=0x7f73e8005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:17:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fa7fc700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.121:3300/0 conn(0x7f73e8003730 0x7f73e8005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fa7fc700 1 -- 172.21.15.94:0/3956885928 >> v1:172.21.15.94:6789/0 conn(0x7f73e80061b0 legacy=0x7f73e8008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fa7fc700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73e80102d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fa7fc700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73e8012820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f7401107700 1 -- 172.21.15.94:0/3956885928 reap_dead start 2023-09-30T00:17:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fa7fc700 1 -- 172.21.15.94:0/3956885928 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e800afb0 con 0x7f73e800a020 2023-09-30T00:17:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fa7fc700 1 -- 172.21.15.94:0/3956885928 --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73fc06ae10 con 0x7f73fc06e0d0 2023-09-30T00:17:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fa7fc700 1 -- 172.21.15.94:0/3956885928 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0a60d0 con 0x7f73fc0a6900 2023-09-30T00:17:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f7401107700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73e8012820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73e80102d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fbfff700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 msgr2=0x7f73e80102d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:17:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fbfff700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 msgr2=0x7f73e80102d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f7401107700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73e8012820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T00:17:34.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f7401107700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73e8012820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:17:34.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73e80102d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:34.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f7401107700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73e8012820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:17:34.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f7401107700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73e8012820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:17:34.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.183+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73e80102d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:17:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.363+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241551338 0 0) 0x7f73e800afb0 con 0x7f73e800a020 2023-09-30T00:17:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.363+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e4003570 con 0x7f73e800a020 2023-09-30T00:17:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2119898041 0 0) 0x7f73e4003570 con 0x7f73e800a020 2023-09-30T00:17:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73e8012820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:17:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73e8012820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 msgr2=0x7f73e80102d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:17:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 --2- 172.21.15.94:0/3956885928 >> v2:172.21.15.94:3300/0 conn(0x7f73fc06e0d0 0x7f73e80102d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e8008e30 con 0x7f73e800a020 2023-09-30T00:17:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3153706020 0 0) 0x7f73ec009720 con 0x7f73e800a020 2023-09-30T00:17:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 >> v1:172.21.15.94:6789/0 conn(0x7f73e800a020 legacy=0x7f73e800a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73e4003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73e4004090 0x7f73e4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73e4004d50 0x7f73e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc06ae10 con 0x7f73e4004d50 2023-09-30T00:17:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0a60d0 con 0x7f73e4004090 2023-09-30T00:17:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e400bbf0 con 0x7f73fc0a6900 2023-09-30T00:17:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f7400906700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73e4003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f73fbfff700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73e4004090 0x7f73e4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.364+0000 7f7401107700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73e4004d50 0x7f73e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:34.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.365+0000 7f7401107700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 msgr2=0x7f73e4003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.365+0000 7f7401107700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc0a6900 0x7f73e4003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.365+0000 7f7401107700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73e4004090 msgr2=0x7f73e4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.365+0000 7f7401107700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73e4004090 0x7f73e4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.365+0000 7f7401107700 1 -- 172.21.15.94:0/3956885928 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e4003570 con 0x7f73e4004d50 2023-09-30T00:17:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.365+0000 7f7401107700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73e4004d50 0x7f73e40051c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f73fc069c10 tx=0x7f73f4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612b4cc83ed035ba server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.365+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f40070d0 con 0x7f73e4004d50 2023-09-30T00:17:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.365+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f4007230 con 0x7f73e4004d50 2023-09-30T00:17:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.365+0000 7f73fb7fe700 1 -- 172.21.15.94:0/3956885928 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f40075f0 con 0x7f73e4004d50 2023-09-30T00:17:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.366+0000 7f7401908700 1 -- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73e4004d50 msgr2=0x7f73e40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.366+0000 7f7401908700 1 --2- 172.21.15.94:0/3956885928 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73e4004d50 0x7f73e40051c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f73fc069c10 tx=0x7f73f4009040 comp rx=0 tx=0).stop 2023-09-30T00:17:34.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.366+0000 7f7401107700 1 -- 172.21.15.94:0/3956885928 reap_dead start 2023-09-30T00:17:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.366+0000 7f7401908700 1 -- 172.21.15.94:0/3956885928 shutdown_connections 2023-09-30T00:17:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.366+0000 7f7401908700 1 -- 172.21.15.94:0/3956885928 >> 172.21.15.94:0/3956885928 conn(0x7f73fc0b43f0 msgr2=0x7f73fc06c760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.366+0000 7f7401908700 1 -- 172.21.15.94:0/3956885928 shutdown_connections 2023-09-30T00:17:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.366+0000 7f7401908700 1 -- 172.21.15.94:0/3956885928 wait complete. 2023-09-30T00:17:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.367+0000 7f7401908700 1 Processor -- start 2023-09-30T00:17:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.368+0000 7f7401908700 1 -- start start 2023-09-30T00:17:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7401908700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73fc09f880 0x7f73fc096360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7401908700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73fc09e2a0 0x7f73fc09e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7401908700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc13b0c0 0x7f73fc13b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7401908700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0a3780 con 0x7f73fc09e2a0 2023-09-30T00:17:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7401908700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc09ef30 con 0x7f73fc09f880 2023-09-30T00:17:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7401908700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc09f1f0 con 0x7f73fc13b0c0 2023-09-30T00:17:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f73fbfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73fc09e2a0 0x7f73fc09e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f73fbfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73fc09e2a0 0x7f73fc09e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46432/0 (socket says 172.21.15.94:46432) 2023-09-30T00:17:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f73fbfff700 1 -- 172.21.15.94:0/2681384767 learned_addr learned my addr 172.21.15.94:0/2681384767 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7400906700 1 --2- 172.21.15.94:0/2681384767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73fc09f880 0x7f73fc096360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7400906700 1 -- 172.21.15.94:0/2681384767 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc13b0c0 msgr2=0x7f73fc13b530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T00:17:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7400906700 1 --2- 172.21.15.94:0/2681384767 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc13b0c0 0x7f73fc13b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7400906700 1 -- 172.21.15.94:0/2681384767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73fc09e2a0 msgr2=0x7f73fc09e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7400906700 1 --2- 172.21.15.94:0/2681384767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73fc09e2a0 0x7f73fc09e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7400906700 1 -- 172.21.15.94:0/2681384767 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc09ca00 con 0x7f73fc09f880 2023-09-30T00:17:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f73fbfff700 1 --2- 172.21.15.94:0/2681384767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73fc09e2a0 0x7f73fc09e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.369+0000 7f7400906700 1 --2- 172.21.15.94:0/2681384767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73fc09f880 0x7f73fc096360 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0070b0 tx=0x7f73ec00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.371+0000 7f73f8ff9700 1 -- 172.21.15.94:0/2681384767 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec020600 con 0x7f73fc09f880 2023-09-30T00:17:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.371+0000 7f7401908700 1 -- 172.21.15.94:0/2681384767 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73fc09fc50 con 0x7f73fc09f880 2023-09-30T00:17:34.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.371+0000 7f7401908700 1 -- 172.21.15.94:0/2681384767 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73fc09d3d0 con 0x7f73fc09f880 2023-09-30T00:17:34.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.372+0000 7f73f8ff9700 1 -- 172.21.15.94:0/2681384767 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ec01a070 con 0x7f73fc09f880 2023-09-30T00:17:34.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.372+0000 7f73f8ff9700 1 -- 172.21.15.94:0/2681384767 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec0209b0 con 0x7f73fc09f880 2023-09-30T00:17:34.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.373+0000 7f7401908700 1 -- 172.21.15.94:0/2681384767 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73dc0050b0 con 0x7f73fc09f880 2023-09-30T00:17:34.378 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.376+0000 7f73f8ff9700 1 -- 172.21.15.94:0/2681384767 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f73ec032420 con 0x7f73fc09f880 2023-09-30T00:17:34.378 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.376+0000 7f73f8ff9700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-30T00:17:34.378 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.376+0000 7f73f8ff9700 4 mgrc handle_mgr_map Got map version 1 2023-09-30T00:17:34.378 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.376+0000 7f73f8ff9700 4 mgrc handle_mgr_map Active mgr is now 2023-09-30T00:17:34.378 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.376+0000 7f73f8ff9700 4 mgrc reconnect No active mgr available yet 2023-09-30T00:17:34.378 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.376+0000 7f73f8ff9700 1 -- 172.21.15.94:0/2681384767 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f73ec029750 con 0x7f73fc09f880 2023-09-30T00:17:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.379+0000 7f73f8ff9700 1 -- 172.21.15.94:0/2681384767 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f73ec020da0 con 0x7f73fc09f880 2023-09-30T00:17:34.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.457+0000 7f7401908700 1 -- 172.21.15.94:0/2681384767 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f73dc005630 con 0x7f73fc09f880 2023-09-30T00:17:34.461 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.460+0000 7f73f8ff9700 1 -- 172.21.15.94:0/2681384767 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f73ec036090 con 0x7f73fc09f880 2023-09-30T00:17:34.463 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.461+0000 7f7401908700 10 mgrc shutdown 2023-09-30T00:17:34.463 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.461+0000 7f7401908700 1 -- 172.21.15.94:0/2681384767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73fc09f880 msgr2=0x7f73fc096360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:34.463 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.461+0000 7f7401908700 1 --2- 172.21.15.94:0/2681384767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73fc09f880 0x7f73fc096360 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0070b0 tx=0x7f73ec00b040 comp rx=0 tx=0).stop 2023-09-30T00:17:34.463 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.461+0000 7f7401908700 1 -- 172.21.15.94:0/2681384767 shutdown_connections 2023-09-30T00:17:34.463 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.461+0000 7f7401908700 1 --2- 172.21.15.94:0/2681384767 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73fc13b0c0 0x7f73fc13b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.463 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.461+0000 7f7401908700 1 --2- 172.21.15.94:0/2681384767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73fc09f880 0x7f73fc096360 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.463 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.461+0000 7f7401908700 1 --2- 172.21.15.94:0/2681384767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73fc09e2a0 0x7f73fc09e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.464 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.461+0000 7f7401908700 1 -- 172.21.15.94:0/2681384767 >> 172.21.15.94:0/2681384767 conn(0x7f73fc0b43f0 msgr2=0x7f73fc06d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:34.464 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.461+0000 7f7401908700 1 -- 172.21.15.94:0/2681384767 shutdown_connections 2023-09-30T00:17:34.464 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:34.461+0000 7f7401908700 1 -- 172.21.15.94:0/2681384767 wait complete. 2023-09-30T00:17:34.464 INFO:teuthology.orchestra.run.smithi094.stderr:adjusted tunables profile to default 2023-09-30T00:17:34.475 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T00:17:34.475 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-09-30T00:17:34.476 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T00:17:34.501 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:34.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T00:17:34.526 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:34.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T00:17:34.592 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-09-30T00:17:34.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T00:17:34.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new c508b402-f8ce-4461-823b-9e35fcd8d40f 0 2023-09-30T00:17:34.816 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.813+0000 7f4d2db68700 1 Processor -- start 2023-09-30T00:17:34.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.813+0000 7f4d2db68700 1 -- start start 2023-09-30T00:17:34.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.813+0000 7f4d2db68700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d2809e6d0 0x7f4d28142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.813+0000 7f4d2db68700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4d28141b60 0x7f4d281346b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d2db68700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d280a5b10 con 0x7f4d280b50f0 2023-09-30T00:17:34.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d2db68700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d280a5dd0 con 0x7f4d28141b60 2023-09-30T00:17:34.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d2db68700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2809eaa0 con 0x7f4d2809e6d0 2023-09-30T00:17:34.817 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d27fff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4d280b50f0 0x7f4d2806dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:37262/0 (socket says 172.21.15.121:37262) 2023-09-30T00:17:34.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d27fff700 1 -- 172.21.15.121:0/2820246774 learned_addr learned my addr 172.21.15.121:0/2820246774 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-30T00:17:34.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d26ffd700 1 --2- 172.21.15.121:0/2820246774 >> v2:172.21.15.94:3300/0 conn(0x7f4d28141b60 0x7f4d281346b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:34.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d277fe700 1 --2- 172.21.15.121:0/2820246774 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d2809e6d0 0x7f4d28142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:34.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d267fc700 1 -- 172.21.15.121:0/2820246774 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005310965 0 0) 0x7f4d280a5b10 con 0x7f4d280b50f0 2023-09-30T00:17:34.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d267fc700 1 -- 172.21.15.121:0/2820246774 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d10003570 con 0x7f4d280b50f0 2023-09-30T00:17:34.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d26ffd700 1 -- 172.21.15.121:0/2820246774 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d2809e6d0 msgr2=0x7f4d28142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:34.818 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d26ffd700 1 --2- 172.21.15.121:0/2820246774 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d2809e6d0 0x7f4d28142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d26ffd700 1 -- 172.21.15.121:0/2820246774 >> v1:172.21.15.121:6789/0 conn(0x7f4d280b50f0 legacy=0x7f4d2806dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d26ffd700 1 -- 172.21.15.121:0/2820246774 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d280a5740 con 0x7f4d28141b60 2023-09-30T00:17:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d277fe700 1 --2- 172.21.15.121:0/2820246774 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d2809e6d0 0x7f4d28142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.814+0000 7f4d26ffd700 1 --2- 172.21.15.121:0/2820246774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d28141b60 0x7f4d281346b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c0098e0 tx=0x7f4d1c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdb28973d210a9c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.815+0000 7f4d267fc700 1 -- 172.21.15.121:0/2820246774 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c011630 con 0x7f4d28141b60 2023-09-30T00:17:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.815+0000 7f4d267fc700 1 -- 172.21.15.121:0/2820246774 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1c011790 con 0x7f4d28141b60 2023-09-30T00:17:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.815+0000 7f4d267fc700 1 -- 172.21.15.121:0/2820246774 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c011b30 con 0x7f4d28141b60 2023-09-30T00:17:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.815+0000 7f4d2db68700 1 -- 172.21.15.121:0/2820246774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d28141b60 msgr2=0x7f4d281346b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:34.819 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.815+0000 7f4d2db68700 1 --2- 172.21.15.121:0/2820246774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d28141b60 0x7f4d281346b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c0098e0 tx=0x7f4d1c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.815+0000 7f4d2db68700 1 -- 172.21.15.121:0/2820246774 shutdown_connections 2023-09-30T00:17:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.815+0000 7f4d2db68700 1 --2- 172.21.15.121:0/2820246774 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d2809e6d0 0x7f4d28142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.815+0000 7f4d2db68700 1 --2- 172.21.15.121:0/2820246774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d28141b60 0x7f4d281346b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.815+0000 7f4d2db68700 1 -- 172.21.15.121:0/2820246774 >> 172.21.15.121:0/2820246774 conn(0x7f4d2806a940 msgr2=0x7f4d2806ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.815+0000 7f4d2db68700 1 -- 172.21.15.121:0/2820246774 shutdown_connections 2023-09-30T00:17:34.820 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.815+0000 7f4d2db68700 1 -- 172.21.15.121:0/2820246774 wait complete. 2023-09-30T00:17:34.822 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.816+0000 7f4d2db68700 1 Processor -- start 2023-09-30T00:17:34.822 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.816+0000 7f4d2db68700 1 -- start start 2023-09-30T00:17:34.822 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d2db68700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2809e6d0 0x7f4d28098030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d2db68700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d280b50f0 0x7f4d280a3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d2db68700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d28141b60 0x7f4d2814ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d2db68700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2814e630 con 0x7f4d280b50f0 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d2db68700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2814e8d0 con 0x7f4d2809e6d0 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d2db68700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d280b2450 con 0x7f4d28141b60 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d277fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2809e6d0 0x7f4d28098030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d277fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2809e6d0 0x7f4d28098030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50686/0 (socket says 172.21.15.121:50686) 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d277fe700 1 -- 172.21.15.121:0/2203564140 learned_addr learned my addr 172.21.15.121:0/2203564140 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d27fff700 1 --2- 172.21.15.121:0/2203564140 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d28141b60 0x7f4d2814ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d26ffd700 1 --2- 172.21.15.121:0/2203564140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d280b50f0 0x7f4d280a3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d277fe700 1 -- 172.21.15.121:0/2203564140 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d28141b60 msgr2=0x7f4d2814ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d277fe700 1 --2- 172.21.15.121:0/2203564140 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d28141b60 0x7f4d2814ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.823 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d277fe700 1 -- 172.21.15.121:0/2203564140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d280b50f0 msgr2=0x7f4d280a3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d277fe700 1 --2- 172.21.15.121:0/2203564140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d280b50f0 0x7f4d280a3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d277fe700 1 -- 172.21.15.121:0/2203564140 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d18009420 con 0x7f4d2809e6d0 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.817+0000 7f4d277fe700 1 --2- 172.21.15.121:0/2203564140 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2809e6d0 0x7f4d28098030 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800ec80 tx=0x7f4d180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.818+0000 7f4d0f7fe700 1 -- 172.21.15.121:0/2203564140 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1800c850 con 0x7f4d2809e6d0 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.818+0000 7f4d0f7fe700 1 -- 172.21.15.121:0/2203564140 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1800c9b0 con 0x7f4d2809e6d0 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.818+0000 7f4d0f7fe700 1 -- 172.21.15.121:0/2203564140 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1800cd30 con 0x7f4d2809e6d0 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.818+0000 7f4d2db68700 1 -- 172.21.15.121:0/2203564140 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1c009420 con 0x7f4d2809e6d0 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.818+0000 7f4d2db68700 1 -- 172.21.15.121:0/2203564140 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2814e270 con 0x7f4d2809e6d0 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.818+0000 7f4d0f7fe700 1 -- 172.21.15.121:0/2203564140 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4d18013900 con 0x7f4d2809e6d0 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.818+0000 7f4d0f7fe700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.818+0000 7f4d0f7fe700 4 mgrc handle_mgr_map Got map version 1 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.818+0000 7f4d0f7fe700 4 mgrc handle_mgr_map Active mgr is now 2023-09-30T00:17:34.824 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.818+0000 7f4d0f7fe700 4 mgrc reconnect No active mgr available yet 2023-09-30T00:17:34.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.818+0000 7f4d0f7fe700 1 -- 172.21.15.121:0/2203564140 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4d1801c970 con 0x7f4d2809e6d0 2023-09-30T00:17:34.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.818+0000 7f4d2db68700 1 -- 172.21.15.121:0/2203564140 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d280a0330 con 0x7f4d2809e6d0 2023-09-30T00:17:34.825 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.821+0000 7f4d0f7fe700 1 -- 172.21.15.121:0/2203564140 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4d18021060 con 0x7f4d2809e6d0 2023-09-30T00:17:34.904 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.903+0000 7f4d2db68700 1 -- 172.21.15.121:0/2203564140 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c508b402-f8ce-4461-823b-9e35fcd8d40f", "id": 0} v 0) v1 -- 0x7f4d280b4c50 con 0x7f4d2809e6d0 2023-09-30T00:17:34.908 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.907+0000 7f4d0f7fe700 1 -- 172.21.15.121:0/2203564140 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c508b402-f8ce-4461-823b-9e35fcd8d40f", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4d1802a0b0 con 0x7f4d2809e6d0 2023-09-30T00:17:34.908 INFO:teuthology.orchestra.run.smithi121.stdout:0 2023-09-30T00:17:34.910 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.908+0000 7f4d2db68700 10 mgrc shutdown 2023-09-30T00:17:34.911 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.908+0000 7f4d2db68700 1 -- 172.21.15.121:0/2203564140 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2809e6d0 msgr2=0x7f4d28098030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:34.911 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.908+0000 7f4d2db68700 1 --2- 172.21.15.121:0/2203564140 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2809e6d0 0x7f4d28098030 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800ec80 tx=0x7f4d180091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.911 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.908+0000 7f4d2db68700 1 -- 172.21.15.121:0/2203564140 shutdown_connections 2023-09-30T00:17:34.911 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.908+0000 7f4d2db68700 1 --2- 172.21.15.121:0/2203564140 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d28141b60 0x7f4d2814ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.911 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.908+0000 7f4d2db68700 1 --2- 172.21.15.121:0/2203564140 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2809e6d0 0x7f4d28098030 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.911 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.908+0000 7f4d2db68700 1 --2- 172.21.15.121:0/2203564140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d280b50f0 0x7f4d280a3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:34.912 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.908+0000 7f4d2db68700 1 -- 172.21.15.121:0/2203564140 >> 172.21.15.121:0/2203564140 conn(0x7f4d2806a940 msgr2=0x7f4d2813ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:34.912 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.908+0000 7f4d2db68700 1 -- 172.21.15.121:0/2203564140 shutdown_connections 2023-09-30T00:17:34.912 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:34.908+0000 7f4d2db68700 1 -- 172.21.15.121:0/2203564140 wait complete. 2023-09-30T00:17:34.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new e96ecdf0-bb38-4eb1-be99-54f813c12888 1 2023-09-30T00:17:35.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.036+0000 7f75a3a40700 1 Processor -- start 2023-09-30T00:17:35.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.036+0000 7f75a3a40700 1 -- start start 2023-09-30T00:17:35.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a3a40700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f759c0702f0 0x7f759c09ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a3a40700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f759c09d360 0x7f759c09d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a3a40700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c09dce0 con 0x7f759c0706c0 2023-09-30T00:17:35.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a3a40700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759c09df80 con 0x7f759c0702f0 2023-09-30T00:17:35.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a3a40700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759c13d540 con 0x7f759c09d360 2023-09-30T00:17:35.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a1fdd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f759c09d360 0x7f759c09d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a1fdd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f759c09d360 0x7f759c09d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50688/0 (socket says 172.21.15.121:50688) 2023-09-30T00:17:35.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a0fdb700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f759c0706c0 0x7f759c13ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:37274/0 (socket says 172.21.15.121:37274) 2023-09-30T00:17:35.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a1fdd700 1 -- 172.21.15.121:0/928136685 learned_addr learned my addr 172.21.15.121:0/928136685 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-30T00:17:35.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a17dc700 1 --2- 172.21.15.121:0/928136685 >> v2:172.21.15.94:3300/0 conn(0x7f759c0702f0 0x7f759c09ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a1fdd700 1 -- 172.21.15.121:0/928136685 >> v2:172.21.15.94:3300/0 conn(0x7f759c0702f0 msgr2=0x7f759c09ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:35.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a1fdd700 1 --2- 172.21.15.121:0/928136685 >> v2:172.21.15.94:3300/0 conn(0x7f759c0702f0 0x7f759c09ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a1fdd700 1 -- 172.21.15.121:0/928136685 >> v1:172.21.15.121:6789/0 conn(0x7f759c0706c0 legacy=0x7f759c13ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a1fdd700 1 -- 172.21.15.121:0/928136685 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c13d180 con 0x7f759c09d360 2023-09-30T00:17:35.040 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f75a1fdd700 1 --2- 172.21.15.121:0/928136685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f759c09d360 0x7f759c09d7a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f759c13e470 tx=0x7f7594009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79e5188b77ca401a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:35.041 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.037+0000 7f7593fff700 1 -- 172.21.15.121:0/928136685 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940092e0 con 0x7f759c09d360 2023-09-30T00:17:35.041 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.038+0000 7f7593fff700 1 -- 172.21.15.121:0/928136685 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759400ba20 con 0x7f759c09d360 2023-09-30T00:17:35.041 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.038+0000 7f7593fff700 1 -- 172.21.15.121:0/928136685 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759400bde0 con 0x7f759c09d360 2023-09-30T00:17:35.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.038+0000 7f75a3a40700 1 -- 172.21.15.121:0/928136685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f759c09d360 msgr2=0x7f759c09d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.038+0000 7f75a3a40700 1 --2- 172.21.15.121:0/928136685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f759c09d360 0x7f759c09d7a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f759c13e470 tx=0x7f7594009840 comp rx=0 tx=0).stop 2023-09-30T00:17:35.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.038+0000 7f75a3a40700 1 -- 172.21.15.121:0/928136685 shutdown_connections 2023-09-30T00:17:35.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.038+0000 7f75a3a40700 1 --2- 172.21.15.121:0/928136685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f759c09d360 0x7f759c09d7a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.038+0000 7f75a3a40700 1 --2- 172.21.15.121:0/928136685 >> v2:172.21.15.94:3300/0 conn(0x7f759c0702f0 0x7f759c09ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.038+0000 7f75a3a40700 1 -- 172.21.15.121:0/928136685 >> 172.21.15.121:0/928136685 conn(0x7f759c0a5810 msgr2=0x7f759c0a3780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:35.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.039+0000 7f75a3a40700 1 -- 172.21.15.121:0/928136685 shutdown_connections 2023-09-30T00:17:35.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.039+0000 7f75a3a40700 1 -- 172.21.15.121:0/928136685 wait complete. 2023-09-30T00:17:35.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.040+0000 7f75a3a40700 1 Processor -- start 2023-09-30T00:17:35.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.040+0000 7f75a3a40700 1 -- start start 2023-09-30T00:17:35.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.040+0000 7f75a3a40700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f759c0702f0 0x7f759c06e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.040+0000 7f75a3a40700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c0706c0 0x7f759c0b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.040+0000 7f75a3a40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f759c09d360 0x7f759c13e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.040+0000 7f75a3a40700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c13fb00 con 0x7f759c0706c0 2023-09-30T00:17:35.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.040+0000 7f75a3a40700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c0b1660 con 0x7f759c09d360 2023-09-30T00:17:35.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.040+0000 7f75a3a40700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759c13edb0 con 0x7f759c0702f0 2023-09-30T00:17:35.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.040+0000 7f75a1fdd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f759c09d360 0x7f759c13e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.040+0000 7f75a1fdd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f759c09d360 0x7f759c13e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50698/0 (socket says 172.21.15.121:50698) 2023-09-30T00:17:35.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.040+0000 7f75a1fdd700 1 -- 172.21.15.121:0/712922817 learned_addr learned my addr 172.21.15.121:0/712922817 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-30T00:17:35.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.040+0000 7f75a0fdb700 1 --2- 172.21.15.121:0/712922817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c0706c0 0x7f759c0b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.046 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75a17dc700 1 --2- 172.21.15.121:0/712922817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f759c0702f0 0x7f759c06e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75a1fdd700 1 -- 172.21.15.121:0/712922817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f759c0702f0 msgr2=0x7f759c06e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75a1fdd700 1 --2- 172.21.15.121:0/712922817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f759c0702f0 0x7f759c06e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75a1fdd700 1 -- 172.21.15.121:0/712922817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c0706c0 msgr2=0x7f759c0b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75a1fdd700 1 --2- 172.21.15.121:0/712922817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c0706c0 0x7f759c0b29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75a1fdd700 1 -- 172.21.15.121:0/712922817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7594009ae0 con 0x7f759c09d360 2023-09-30T00:17:35.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75a0fdb700 1 --2- 172.21.15.121:0/712922817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c0706c0 0x7f759c0b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:35.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75a1fdd700 1 --2- 172.21.15.121:0/712922817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f759c09d360 0x7f759c13e6d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f759c13d680 tx=0x7f759401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:35.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75917fa700 1 -- 172.21.15.121:0/712922817 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940032f0 con 0x7f759c09d360 2023-09-30T00:17:35.047 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75917fa700 1 -- 172.21.15.121:0/712922817 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7594022630 con 0x7f759c09d360 2023-09-30T00:17:35.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75917fa700 1 -- 172.21.15.121:0/712922817 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940229d0 con 0x7f759c09d360 2023-09-30T00:17:35.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75a3a40700 1 -- 172.21.15.121:0/712922817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759c13f070 con 0x7f759c09d360 2023-09-30T00:17:35.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75a3a40700 1 -- 172.21.15.121:0/712922817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c0b23d0 con 0x7f759c09d360 2023-09-30T00:17:35.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75917fa700 1 -- 172.21.15.121:0/712922817 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f7594003040 con 0x7f759c09d360 2023-09-30T00:17:35.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75917fa700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-30T00:17:35.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75917fa700 4 mgrc handle_mgr_map Got map version 1 2023-09-30T00:17:35.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75917fa700 4 mgrc handle_mgr_map Active mgr is now 2023-09-30T00:17:35.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.041+0000 7f75917fa700 4 mgrc reconnect No active mgr available yet 2023-09-30T00:17:35.048 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.042+0000 7f75917fa700 1 -- 172.21.15.121:0/712922817 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f759402acf0 con 0x7f759c09d360 2023-09-30T00:17:35.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.042+0000 7f75a3a40700 1 -- 172.21.15.121:0/712922817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759c06fbd0 con 0x7f759c09d360 2023-09-30T00:17:35.049 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.045+0000 7f75917fa700 1 -- 172.21.15.121:0/712922817 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f75940496d0 con 0x7f759c09d360 2023-09-30T00:17:35.129 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.128+0000 7f75a3a40700 1 -- 172.21.15.121:0/712922817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e96ecdf0-bb38-4eb1-be99-54f813c12888", "id": 1} v 0) v1 -- 0x7f759c0b25f0 con 0x7f759c09d360 2023-09-30T00:17:35.134 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.133+0000 7f75917fa700 1 -- 172.21.15.121:0/712922817 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e96ecdf0-bb38-4eb1-be99-54f813c12888", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7594038030 con 0x7f759c09d360 2023-09-30T00:17:35.134 INFO:teuthology.orchestra.run.smithi121.stdout:1 2023-09-30T00:17:35.135 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.133+0000 7f75a3a40700 10 mgrc shutdown 2023-09-30T00:17:35.135 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.133+0000 7f75a3a40700 1 -- 172.21.15.121:0/712922817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f759c09d360 msgr2=0x7f759c13e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.135 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.134+0000 7f75a3a40700 1 --2- 172.21.15.121:0/712922817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f759c09d360 0x7f759c13e6d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f759c13d680 tx=0x7f759401a000 comp rx=0 tx=0).stop 2023-09-30T00:17:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.134+0000 7f75a3a40700 1 -- 172.21.15.121:0/712922817 shutdown_connections 2023-09-30T00:17:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.134+0000 7f75a3a40700 1 --2- 172.21.15.121:0/712922817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f759c0702f0 0x7f759c06e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.134+0000 7f75a3a40700 1 --2- 172.21.15.121:0/712922817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f759c09d360 0x7f759c13e6d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.134+0000 7f75a3a40700 1 --2- 172.21.15.121:0/712922817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c0706c0 0x7f759c0b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.134+0000 7f75a3a40700 1 -- 172.21.15.121:0/712922817 >> 172.21.15.121:0/712922817 conn(0x7f759c0a5810 msgr2=0x7f759c0a07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.134+0000 7f75a3a40700 1 -- 172.21.15.121:0/712922817 shutdown_connections 2023-09-30T00:17:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.134+0000 7f75a3a40700 1 -- 172.21.15.121:0/712922817 wait complete. 2023-09-30T00:17:35.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 68700ae4-764c-4dff-9cf2-5f33132cd0d9 2 2023-09-30T00:17:35.261 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.260+0000 7f5f62902700 1 Processor -- start 2023-09-30T00:17:35.262 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.261+0000 7f5f62902700 1 -- start start 2023-09-30T00:17:35.264 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.261+0000 7f5f62902700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5f5c0702f0 0x7f5f5c09ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.264 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.261+0000 7f5f62902700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c09d360 0x7f5f5c09d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.264 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.261+0000 7f5f62902700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c09dd20 con 0x7f5f5c0706c0 2023-09-30T00:17:35.264 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.261+0000 7f5f62902700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f5c09df80 con 0x7f5f5c0702f0 2023-09-30T00:17:35.264 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.261+0000 7f5f62902700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c13d540 con 0x7f5f5c09d360 2023-09-30T00:17:35.264 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.261+0000 7f5f5bfff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5f5c0702f0 0x7f5f5c09ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.261+0000 7f5f60e9f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c09d360 0x7f5f5c09d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.261+0000 7f5f5b7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5f5c0706c0 0x7f5f5c13ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.121:58688/0 (socket says 172.21.15.121:58688) 2023-09-30T00:17:35.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.261+0000 7f5f5bfff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5f5c0702f0 0x7f5f5c09ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.121:59348/0 (socket says 172.21.15.121:59348) 2023-09-30T00:17:35.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.261+0000 7f5f5b7fe700 1 -- 172.21.15.121:0/1198318679 learned_addr learned my addr 172.21.15.121:0/1198318679 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-30T00:17:35.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.261+0000 7f5f60e9f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c09d360 0x7f5f5c09d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.121:37206/0 (socket says 172.21.15.121:37206) 2023-09-30T00:17:35.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.262+0000 7f5f5bfff700 1 -- 172.21.15.121:0/1198318679 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c09d360 msgr2=0x7f5f5c09d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:35.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.262+0000 7f5f5bfff700 1 --2- 172.21.15.121:0/1198318679 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c09d360 0x7f5f5c09d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.262+0000 7f5f5bfff700 1 -- 172.21.15.121:0/1198318679 >> v1:172.21.15.94:6789/0 conn(0x7f5f5c0706c0 legacy=0x7f5f5c13ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.262+0000 7f5f5bfff700 1 -- 172.21.15.121:0/1198318679 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c13d180 con 0x7f5f5c0702f0 2023-09-30T00:17:35.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.262+0000 7f5f60e9f700 1 --2- 172.21.15.121:0/1198318679 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c09d360 0x7f5f5c09d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:35.265 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.262+0000 7f5f5bfff700 1 --2- 172.21.15.121:0/1198318679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5c0702f0 0x7f5f5c09ec50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0098e0 tx=0x7f5f4c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5feea29858307295 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:35.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.262+0000 7f5f5affd700 1 -- 172.21.15.121:0/1198318679 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c00b640 con 0x7f5f5c0702f0 2023-09-30T00:17:35.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.262+0000 7f5f5affd700 1 -- 172.21.15.121:0/1198318679 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c00b7a0 con 0x7f5f5c0702f0 2023-09-30T00:17:35.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.262+0000 7f5f5affd700 1 -- 172.21.15.121:0/1198318679 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c00bb40 con 0x7f5f5c0702f0 2023-09-30T00:17:35.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.263+0000 7f5f62902700 1 -- 172.21.15.121:0/1198318679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5c0702f0 msgr2=0x7f5f5c09ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.263+0000 7f5f62902700 1 --2- 172.21.15.121:0/1198318679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5c0702f0 0x7f5f5c09ec50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0098e0 tx=0x7f5f4c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.263+0000 7f5f62902700 1 -- 172.21.15.121:0/1198318679 shutdown_connections 2023-09-30T00:17:35.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.263+0000 7f5f62902700 1 --2- 172.21.15.121:0/1198318679 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c09d360 0x7f5f5c09d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.263+0000 7f5f62902700 1 --2- 172.21.15.121:0/1198318679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5c0702f0 0x7f5f5c09ec50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.263+0000 7f5f62902700 1 -- 172.21.15.121:0/1198318679 >> 172.21.15.121:0/1198318679 conn(0x7f5f5c0a5810 msgr2=0x7f5f5c0a3780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:35.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.263+0000 7f5f62902700 1 -- 172.21.15.121:0/1198318679 shutdown_connections 2023-09-30T00:17:35.266 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.263+0000 7f5f62902700 1 -- 172.21.15.121:0/1198318679 wait complete. 2023-09-30T00:17:35.269 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.264+0000 7f5f62902700 1 Processor -- start 2023-09-30T00:17:35.269 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.264+0000 7f5f62902700 1 -- start start 2023-09-30T00:17:35.269 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.264+0000 7f5f62902700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c0702f0 0x7f5f5c134460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.269 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.264+0000 7f5f62902700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0706c0 0x7f5f5c140650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.269 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.264+0000 7f5f62902700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5c09d360 0x7f5f5c14fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.264+0000 7f5f62902700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c154bd0 con 0x7f5f5c09d360 2023-09-30T00:17:35.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.264+0000 7f5f62902700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c140fb0 con 0x7f5f5c0706c0 2023-09-30T00:17:35.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.264+0000 7f5f62902700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c153d30 con 0x7f5f5c0702f0 2023-09-30T00:17:35.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f5b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0706c0 0x7f5f5c140650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f5b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0706c0 0x7f5f5c140650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50704/0 (socket says 172.21.15.121:50704) 2023-09-30T00:17:35.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f5b7fe700 1 -- 172.21.15.121:0/4256459431 learned_addr learned my addr 172.21.15.121:0/4256459431 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-30T00:17:35.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f5bfff700 1 --2- 172.21.15.121:0/4256459431 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c0702f0 0x7f5f5c134460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f60e9f700 1 --2- 172.21.15.121:0/4256459431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5c09d360 0x7f5f5c14fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f5bfff700 1 -- 172.21.15.121:0/4256459431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0706c0 msgr2=0x7f5f5c140650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f5bfff700 1 --2- 172.21.15.121:0/4256459431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0706c0 0x7f5f5c140650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f5bfff700 1 -- 172.21.15.121:0/4256459431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5c09d360 msgr2=0x7f5f5c14fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f5bfff700 1 --2- 172.21.15.121:0/4256459431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5c09d360 0x7f5f5c14fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f5bfff700 1 -- 172.21.15.121:0/4256459431 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f54009a20 con 0x7f5f5c0702f0 2023-09-30T00:17:35.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f5b7fe700 1 --2- 172.21.15.121:0/4256459431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0706c0 0x7f5f5c140650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:35.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f5bfff700 1 --2- 172.21.15.121:0/4256459431 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c0702f0 0x7f5f5c134460 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c070af0 tx=0x7f5f4c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:35.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f43fff700 1 -- 172.21.15.121:0/4256459431 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c01bbe0 con 0x7f5f5c0702f0 2023-09-30T00:17:35.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f43fff700 1 -- 172.21.15.121:0/4256459431 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c01bd40 con 0x7f5f5c0702f0 2023-09-30T00:17:35.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f62902700 1 -- 172.21.15.121:0/4256459431 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4c009420 con 0x7f5f5c0702f0 2023-09-30T00:17:35.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f43fff700 1 -- 172.21.15.121:0/4256459431 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c011670 con 0x7f5f5c0702f0 2023-09-30T00:17:35.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.265+0000 7f5f62902700 1 -- 172.21.15.121:0/4256459431 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5c141520 con 0x7f5f5c0702f0 2023-09-30T00:17:35.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.266+0000 7f5f43fff700 1 -- 172.21.15.121:0/4256459431 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5f4c011ca0 con 0x7f5f5c0702f0 2023-09-30T00:17:35.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.266+0000 7f5f43fff700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-30T00:17:35.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.266+0000 7f5f43fff700 4 mgrc handle_mgr_map Got map version 1 2023-09-30T00:17:35.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.266+0000 7f5f43fff700 4 mgrc handle_mgr_map Active mgr is now 2023-09-30T00:17:35.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.266+0000 7f5f43fff700 4 mgrc reconnect No active mgr available yet 2023-09-30T00:17:35.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.266+0000 7f5f43fff700 1 -- 172.21.15.121:0/4256459431 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f5f4c02c440 con 0x7f5f5c0702f0 2023-09-30T00:17:35.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.267+0000 7f5f62902700 1 -- 172.21.15.121:0/4256459431 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5c141710 con 0x7f5f5c0702f0 2023-09-30T00:17:35.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.269+0000 7f5f43fff700 1 -- 172.21.15.121:0/4256459431 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5f4c0436d0 con 0x7f5f5c0702f0 2023-09-30T00:17:35.353 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.352+0000 7f5f62902700 1 -- 172.21.15.121:0/4256459431 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd new", "uuid": "68700ae4-764c-4dff-9cf2-5f33132cd0d9", "id": 2} v 0) v1 -- 0x7f5f5c13f460 con 0x7f5f5c0702f0 2023-09-30T00:17:35.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.355+0000 7f5f43fff700 1 -- 172.21.15.121:0/4256459431 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "68700ae4-764c-4dff-9cf2-5f33132cd0d9", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5f4c032070 con 0x7f5f5c0702f0 2023-09-30T00:17:35.356 INFO:teuthology.orchestra.run.smithi121.stdout:2 2023-09-30T00:17:35.357 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.356+0000 7f5f62902700 10 mgrc shutdown 2023-09-30T00:17:35.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.356+0000 7f5f62902700 1 -- 172.21.15.121:0/4256459431 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c0702f0 msgr2=0x7f5f5c134460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.356+0000 7f5f62902700 1 --2- 172.21.15.121:0/4256459431 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c0702f0 0x7f5f5c134460 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c070af0 tx=0x7f5f4c017000 comp rx=0 tx=0).stop 2023-09-30T00:17:35.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.356+0000 7f5f62902700 1 -- 172.21.15.121:0/4256459431 shutdown_connections 2023-09-30T00:17:35.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.356+0000 7f5f62902700 1 --2- 172.21.15.121:0/4256459431 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f5c0702f0 0x7f5f5c134460 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.356+0000 7f5f62902700 1 --2- 172.21.15.121:0/4256459431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5c09d360 0x7f5f5c14fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.356+0000 7f5f62902700 1 --2- 172.21.15.121:0/4256459431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0706c0 0x7f5f5c140650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.356+0000 7f5f62902700 1 -- 172.21.15.121:0/4256459431 >> 172.21.15.121:0/4256459431 conn(0x7f5f5c0a5810 msgr2=0x7f5f5c0b3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:35.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.356+0000 7f5f62902700 1 -- 172.21.15.121:0/4256459431 shutdown_connections 2023-09-30T00:17:35.358 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.356+0000 7f5f62902700 1 -- 172.21.15.121:0/4256459431 wait complete. 2023-09-30T00:17:35.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 54f17ec8-2561-42e8-bbd7-262130ffdac0 3 2023-09-30T00:17:35.491 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.490+0000 7fcc43fff700 1 Processor -- start 2023-09-30T00:17:35.492 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.490+0000 7fcc43fff700 1 -- start start 2023-09-30T00:17:35.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc43fff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcc440702f0 0x7fcc4409efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc43fff700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4409fb30 con 0x7fcc4413c860 2023-09-30T00:17:35.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc43fff700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4409f510 con 0x7fcc440706c0 2023-09-30T00:17:35.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc43fff700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4413d510 con 0x7fcc440702f0 2023-09-30T00:17:35.494 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc427fc700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcc440706c0 0x7fcc4409d590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:37282/0 (socket says 172.21.15.121:37282) 2023-09-30T00:17:35.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc427fc700 1 -- 172.21.15.121:0/816332746 learned_addr learned my addr 172.21.15.121:0/816332746 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-09-30T00:17:35.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc42ffd700 1 --2- 172.21.15.121:0/816332746 >> v2:172.21.15.94:3300/0 conn(0x7fcc440702f0 0x7fcc4409efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc41ffb700 1 -- 172.21.15.121:0/816332746 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057538934 0 0) 0x7fcc4409f510 con 0x7fcc440706c0 2023-09-30T00:17:35.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc41ffb700 1 -- 172.21.15.121:0/816332746 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2c003570 con 0x7fcc440706c0 2023-09-30T00:17:35.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc41ffb700 1 -- 172.21.15.121:0/816332746 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496885618 0 0) 0x7fcc4409fb30 con 0x7fcc4413c860 2023-09-30T00:17:35.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc41ffb700 1 -- 172.21.15.121:0/816332746 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc4409f510 con 0x7fcc4413c860 2023-09-30T00:17:35.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc42ffd700 1 -- 172.21.15.121:0/816332746 >> v1:172.21.15.121:6789/0 conn(0x7fcc440706c0 legacy=0x7fcc4409d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.491+0000 7fcc42ffd700 1 -- 172.21.15.121:0/816332746 >> v1:172.21.15.94:6789/0 conn(0x7fcc4413c860 legacy=0x7fcc4413cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.495 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.492+0000 7fcc42ffd700 1 -- 172.21.15.121:0/816332746 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4413d7d0 con 0x7fcc440702f0 2023-09-30T00:17:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.492+0000 7fcc42ffd700 1 --2- 172.21.15.121:0/816332746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc440702f0 0x7fcc4409efd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcc4413d9c0 tx=0x7fcc340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3abff37f351760eb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.492+0000 7fcc41ffb700 1 -- 172.21.15.121:0/816332746 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34011630 con 0x7fcc440702f0 2023-09-30T00:17:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.492+0000 7fcc41ffb700 1 -- 172.21.15.121:0/816332746 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc34011790 con 0x7fcc440702f0 2023-09-30T00:17:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.492+0000 7fcc41ffb700 1 -- 172.21.15.121:0/816332746 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34011b50 con 0x7fcc440702f0 2023-09-30T00:17:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.493+0000 7fcc43fff700 1 -- 172.21.15.121:0/816332746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc440702f0 msgr2=0x7fcc4409efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.493+0000 7fcc43fff700 1 --2- 172.21.15.121:0/816332746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc440702f0 0x7fcc4409efd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcc4413d9c0 tx=0x7fcc340091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.493+0000 7fcc43fff700 1 -- 172.21.15.121:0/816332746 shutdown_connections 2023-09-30T00:17:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.493+0000 7fcc43fff700 1 --2- 172.21.15.121:0/816332746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc440702f0 0x7fcc4409efd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.493+0000 7fcc43fff700 1 -- 172.21.15.121:0/816332746 >> 172.21.15.121:0/816332746 conn(0x7fcc440a3570 msgr2=0x7fcc440a3970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.493+0000 7fcc43fff700 1 -- 172.21.15.121:0/816332746 shutdown_connections 2023-09-30T00:17:35.496 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.493+0000 7fcc43fff700 1 -- 172.21.15.121:0/816332746 wait complete. 2023-09-30T00:17:35.499 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.494+0000 7fcc43fff700 1 Processor -- start 2023-09-30T00:17:35.499 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.494+0000 7fcc43fff700 1 -- start start 2023-09-30T00:17:35.499 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.494+0000 7fcc43fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc440702f0 0x7fcc441403c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.500 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.494+0000 7fcc43fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc440706c0 0x7fcc440966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.500 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.494+0000 7fcc43fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcc4413c860 0x7fcc44141b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.500 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.494+0000 7fcc43fff700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4414f150 con 0x7fcc440706c0 2023-09-30T00:17:35.500 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.494+0000 7fcc43fff700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc441416b0 con 0x7fcc440702f0 2023-09-30T00:17:35.500 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.494+0000 7fcc43fff700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc44150080 con 0x7fcc4413c860 2023-09-30T00:17:35.500 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.494+0000 7fcc42ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc440702f0 0x7fcc441403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.500 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.494+0000 7fcc42ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc440702f0 0x7fcc441403c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50712/0 (socket says 172.21.15.121:50712) 2023-09-30T00:17:35.500 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.494+0000 7fcc42ffd700 1 -- 172.21.15.121:0/628934086 learned_addr learned my addr 172.21.15.121:0/628934086 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-09-30T00:17:35.500 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.494+0000 7fcc427fc700 1 --2- 172.21.15.121:0/628934086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc440706c0 0x7fcc440966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.500 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc437fe700 1 --2- 172.21.15.121:0/628934086 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcc4413c860 0x7fcc44141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc427fc700 1 -- 172.21.15.121:0/628934086 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcc4413c860 msgr2=0x7fcc44141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc427fc700 1 --2- 172.21.15.121:0/628934086 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcc4413c860 0x7fcc44141b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc427fc700 1 -- 172.21.15.121:0/628934086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc440702f0 msgr2=0x7fcc441403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc427fc700 1 --2- 172.21.15.121:0/628934086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc440702f0 0x7fcc441403c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc427fc700 1 -- 172.21.15.121:0/628934086 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc34009420 con 0x7fcc440706c0 2023-09-30T00:17:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc42ffd700 1 --2- 172.21.15.121:0/628934086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc440702f0 0x7fcc441403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc427fc700 1 --2- 172.21.15.121:0/628934086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc440706c0 0x7fcc440966e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcc380058d0 tx=0x7fcc38007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc2affd700 1 -- 172.21.15.121:0/628934086 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38005b30 con 0x7fcc440706c0 2023-09-30T00:17:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc2affd700 1 -- 172.21.15.121:0/628934086 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc38005c90 con 0x7fcc440706c0 2023-09-30T00:17:35.501 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc43fff700 1 -- 172.21.15.121:0/628934086 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4414f340 con 0x7fcc440706c0 2023-09-30T00:17:35.502 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc2affd700 1 -- 172.21.15.121:0/628934086 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38007300 con 0x7fcc440706c0 2023-09-30T00:17:35.502 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc43fff700 1 -- 172.21.15.121:0/628934086 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc44142080 con 0x7fcc440706c0 2023-09-30T00:17:35.502 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc2affd700 1 -- 172.21.15.121:0/628934086 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fcc3801c6c0 con 0x7fcc440706c0 2023-09-30T00:17:35.502 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc2affd700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-30T00:17:35.502 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc2affd700 4 mgrc handle_mgr_map Got map version 1 2023-09-30T00:17:35.502 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc2affd700 4 mgrc handle_mgr_map Active mgr is now 2023-09-30T00:17:35.502 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc2affd700 4 mgrc reconnect No active mgr available yet 2023-09-30T00:17:35.502 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.495+0000 7fcc2affd700 1 -- 172.21.15.121:0/628934086 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7fcc38005e00 con 0x7fcc440706c0 2023-09-30T00:17:35.502 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.496+0000 7fcc43fff700 1 -- 172.21.15.121:0/628934086 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc44141860 con 0x7fcc440706c0 2023-09-30T00:17:35.503 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.499+0000 7fcc2affd700 1 -- 172.21.15.121:0/628934086 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fcc38017070 con 0x7fcc440706c0 2023-09-30T00:17:35.550 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:35.549+0000 7fce027ab180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:17:35.582 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.581+0000 7fcc43fff700 1 -- 172.21.15.121:0/628934086 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd new", "uuid": "54f17ec8-2561-42e8-bbd7-262130ffdac0", "id": 3} v 0) v1 -- 0x7fcc4404c380 con 0x7fcc440706c0 2023-09-30T00:17:35.585 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.584+0000 7fcc2affd700 1 -- 172.21.15.121:0/628934086 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "54f17ec8-2561-42e8-bbd7-262130ffdac0", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcc380076f0 con 0x7fcc440706c0 2023-09-30T00:17:35.585 INFO:teuthology.orchestra.run.smithi121.stdout:3 2023-09-30T00:17:35.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.585+0000 7fcc43fff700 10 mgrc shutdown 2023-09-30T00:17:35.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.585+0000 7fcc43fff700 1 -- 172.21.15.121:0/628934086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc440706c0 msgr2=0x7fcc440966e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.586 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.585+0000 7fcc43fff700 1 --2- 172.21.15.121:0/628934086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc440706c0 0x7fcc440966e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcc380058d0 tx=0x7fcc38007d30 comp rx=0 tx=0).stop 2023-09-30T00:17:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.585+0000 7fcc43fff700 1 -- 172.21.15.121:0/628934086 shutdown_connections 2023-09-30T00:17:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.585+0000 7fcc43fff700 1 --2- 172.21.15.121:0/628934086 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcc4413c860 0x7fcc44141b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.585+0000 7fcc43fff700 1 --2- 172.21.15.121:0/628934086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc440702f0 0x7fcc441403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.585+0000 7fcc43fff700 1 --2- 172.21.15.121:0/628934086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc440706c0 0x7fcc440966e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.585+0000 7fcc43fff700 1 -- 172.21.15.121:0/628934086 >> 172.21.15.121:0/628934086 conn(0x7fcc440a3570 msgr2=0x7fcc440b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.585+0000 7fcc43fff700 1 -- 172.21.15.121:0/628934086 shutdown_connections 2023-09-30T00:17:35.587 INFO:teuthology.orchestra.run.smithi121.stderr:2023-09-30T00:17:35.585+0000 7fcc43fff700 1 -- 172.21.15.121:0/628934086 wait complete. 2023-09-30T00:17:35.598 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T00:17:35.598 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-30T00:17:35.600 INFO:tasks.ceph.osd.0:Started 2023-09-30T00:17:35.600 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T00:17:35.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-30T00:17:35.602 INFO:tasks.ceph.osd.1:Started 2023-09-30T00:17:35.602 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T00:17:35.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-30T00:17:35.604 INFO:tasks.ceph.osd.2:Started 2023-09-30T00:17:35.604 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T00:17:35.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-30T00:17:35.606 INFO:tasks.ceph.osd.3:Started 2023-09-30T00:17:35.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:17:35.625 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:35.624+0000 7f67d01f4180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:17:35.640 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:35.639+0000 7f71f7231180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:17:35.687 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:35.686+0000 7fce027ab180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:17:35.724 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.722+0000 7f20474ef700 1 Processor -- start 2023-09-30T00:17:35.724 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.723+0000 7f20474ef700 1 -- start start 2023-09-30T00:17:35.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.723+0000 7f20474ef700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20400b7160 0x7f20400a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.723+0000 7f20474ef700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f204013fd70 0x7f2040140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.723+0000 7f20474ef700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20400a1310 con 0x7f2040070c50 2023-09-30T00:17:35.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.723+0000 7f20474ef700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2040071020 con 0x7f20400b7160 2023-09-30T00:17:35.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.723+0000 7f20474ef700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20400b76a0 con 0x7f204013fd70 2023-09-30T00:17:35.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.723+0000 7f2044a8a700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20400b7160 0x7f20400a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.723+0000 7f204528b700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2040070c50 0x7f20400a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:44050/0 (socket says 172.21.15.94:44050) 2023-09-30T00:17:35.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.723+0000 7f204528b700 1 -- 172.21.15.94:0/2391160375 learned_addr learned my addr 172.21.15.94:0/2391160375 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:35.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.723+0000 7f2045a8c700 1 --2- 172.21.15.94:0/2391160375 >> v2:172.21.15.121:3300/0 conn(0x7f204013fd70 0x7f2040140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.724+0000 7f2044a8a700 1 -- 172.21.15.94:0/2391160375 >> v2:172.21.15.121:3300/0 conn(0x7f204013fd70 msgr2=0x7f2040140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:35.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.724+0000 7f2044a8a700 1 --2- 172.21.15.94:0/2391160375 >> v2:172.21.15.121:3300/0 conn(0x7f204013fd70 0x7f2040140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.724+0000 7f2044a8a700 1 -- 172.21.15.94:0/2391160375 >> v1:172.21.15.94:6789/0 conn(0x7f2040070c50 legacy=0x7f20400a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.724+0000 7f2037fff700 1 -- 172.21.15.94:0/2391160375 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114015189 0 0) 0x7f203c001490 con 0x7f2040070c50 2023-09-30T00:17:35.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.724+0000 7f2044a8a700 1 -- 172.21.15.94:0/2391160375 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20400b7940 con 0x7f20400b7160 2023-09-30T00:17:35.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.724+0000 7f2044a8a700 1 --2- 172.21.15.94:0/2391160375 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20400b7160 0x7f20400a0dd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f20400a31a0 tx=0x7f20300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f70402ea74b0ce6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:35.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.724+0000 7f2037fff700 1 -- 172.21.15.94:0/2391160375 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203000b690 con 0x7f20400b7160 2023-09-30T00:17:35.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.724+0000 7f2037fff700 1 -- 172.21.15.94:0/2391160375 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203000b7f0 con 0x7f20400b7160 2023-09-30T00:17:35.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.724+0000 7f2037fff700 1 -- 172.21.15.94:0/2391160375 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203000bb90 con 0x7f20400b7160 2023-09-30T00:17:35.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.724+0000 7f20474ef700 1 -- 172.21.15.94:0/2391160375 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20400b7160 msgr2=0x7f20400a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.724+0000 7f20474ef700 1 --2- 172.21.15.94:0/2391160375 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20400b7160 0x7f20400a0dd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f20400a31a0 tx=0x7f20300091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.725+0000 7f20474ef700 1 -- 172.21.15.94:0/2391160375 shutdown_connections 2023-09-30T00:17:35.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.725+0000 7f20474ef700 1 --2- 172.21.15.94:0/2391160375 >> v2:172.21.15.121:3300/0 conn(0x7f204013fd70 0x7f2040140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.725+0000 7f20474ef700 1 --2- 172.21.15.94:0/2391160375 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20400b7160 0x7f20400a0dd0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.725+0000 7f20474ef700 1 -- 172.21.15.94:0/2391160375 >> 172.21.15.94:0/2391160375 conn(0x7f204006a090 msgr2=0x7f20400702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:35.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.725+0000 7f20474ef700 1 -- 172.21.15.94:0/2391160375 shutdown_connections 2023-09-30T00:17:35.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.725+0000 7f20474ef700 1 -- 172.21.15.94:0/2391160375 wait complete. 2023-09-30T00:17:35.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.725+0000 7f20474ef700 1 Processor -- start 2023-09-30T00:17:35.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.726+0000 7f20474ef700 1 -- start start 2023-09-30T00:17:35.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.726+0000 7f20474ef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2040070c50 0x7f204013a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.726+0000 7f20474ef700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f20400b7160 0x7f20400a5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.726+0000 7f20474ef700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f204013fd70 0x7f20400a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:35.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.726+0000 7f20474ef700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20400a57e0 con 0x7f204013fd70 2023-09-30T00:17:35.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.726+0000 7f20474ef700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20400a5aa0 con 0x7f2040070c50 2023-09-30T00:17:35.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.726+0000 7f20474ef700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20400a6520 con 0x7f20400b7160 2023-09-30T00:17:35.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.726+0000 7f2045a8c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f204013fd70 0x7f20400a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.726+0000 7f2044a8a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f20400b7160 0x7f20400a5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.726+0000 7f2045a8c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f204013fd70 0x7f20400a4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46474/0 (socket says 172.21.15.94:46474) 2023-09-30T00:17:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.726+0000 7f2045a8c700 1 -- 172.21.15.94:0/558803027 learned_addr learned my addr 172.21.15.94:0/558803027 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.726+0000 7f204528b700 1 --2- 172.21.15.94:0/558803027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2040070c50 0x7f204013a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f2044a8a700 1 -- 172.21.15.94:0/558803027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2040070c50 msgr2=0x7f204013a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f2044a8a700 1 --2- 172.21.15.94:0/558803027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2040070c50 0x7f204013a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f2044a8a700 1 -- 172.21.15.94:0/558803027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f204013fd70 msgr2=0x7f20400a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f2044a8a700 1 --2- 172.21.15.94:0/558803027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f204013fd70 0x7f20400a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f2044a8a700 1 -- 172.21.15.94:0/558803027 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2030009420 con 0x7f20400b7160 2023-09-30T00:17:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f2045a8c700 1 --2- 172.21.15.94:0/558803027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f204013fd70 0x7f20400a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:35.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f2044a8a700 1 --2- 172.21.15.94:0/558803027 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f20400b7160 0x7f20400a5fe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2030016000 tx=0x7f2030017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:35.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f20357fa700 1 -- 172.21.15.94:0/558803027 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203000bb70 con 0x7f20400b7160 2023-09-30T00:17:35.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f20357fa700 1 -- 172.21.15.94:0/558803027 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2030011420 con 0x7f20400b7160 2023-09-30T00:17:35.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f20357fa700 1 -- 172.21.15.94:0/558803027 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20300117c0 con 0x7f20400b7160 2023-09-30T00:17:35.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f20474ef700 1 -- 172.21.15.94:0/558803027 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20400a3da0 con 0x7f20400b7160 2023-09-30T00:17:35.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f20474ef700 1 -- 172.21.15.94:0/558803027 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20401446c0 con 0x7f20400b7160 2023-09-30T00:17:35.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f20357fa700 1 -- 172.21.15.94:0/558803027 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f203002e450 con 0x7f20400b7160 2023-09-30T00:17:35.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f20357fa700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-30T00:17:35.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f20357fa700 4 mgrc handle_mgr_map Got map version 1 2023-09-30T00:17:35.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f20357fa700 4 mgrc handle_mgr_map Active mgr is now 2023-09-30T00:17:35.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f20357fa700 4 mgrc reconnect No active mgr available yet 2023-09-30T00:17:35.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.727+0000 7f20357fa700 1 -- 172.21.15.94:0/558803027 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f2030011e30 con 0x7f20400b7160 2023-09-30T00:17:35.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.728+0000 7f20474ef700 1 -- 172.21.15.94:0/558803027 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20400b7ad0 con 0x7f20400b7160 2023-09-30T00:17:35.734 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.730+0000 7f20357fa700 1 -- 172.21.15.94:0/558803027 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f2030015020 con 0x7f20400b7160 2023-09-30T00:17:35.767 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:35.766+0000 7fce027ab180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:17:35.781 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:35.781+0000 7f71f7231180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:17:35.786 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:35.785+0000 7f67d01f4180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:17:35.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.800+0000 7f20474ef700 1 -- 172.21.15.94:0/558803027 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f204013abe0 con 0x7f20400b7160 2023-09-30T00:17:35.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.801+0000 7f20357fa700 1 -- 172.21.15.94:0/558803027 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) v1 ==== 74+0+4108 (secure 0 0 0) 0x7f204013abe0 con 0x7f20400b7160 2023-09-30T00:17:35.802 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:35.802 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":6,"fsid":"3c891173-587e-4c6f-9c6a-04ec750eedad","created":"2023-09-30T00:17:34.347592+0000","modified":"2023-09-30T00:17:35.583427+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:17:35.583427+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"c508b402-f8ce-4461-823b-9e35fcd8d40f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e96ecdf0-bb38-4eb1-be99-54f813c12888","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"68700ae4-764c-4dff-9cf2-5f33132cd0d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"54f17ec8-2561-42e8-bbd7-262130ffdac0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:17:35.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.801+0000 7f20474ef700 10 mgrc shutdown 2023-09-30T00:17:35.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.802+0000 7f20474ef700 1 -- 172.21.15.94:0/558803027 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f20400b7160 msgr2=0x7f20400a5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:35.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.802+0000 7f20474ef700 1 --2- 172.21.15.94:0/558803027 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f20400b7160 0x7f20400a5fe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2030016000 tx=0x7f2030017000 comp rx=0 tx=0).stop 2023-09-30T00:17:35.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.802+0000 7f20474ef700 1 -- 172.21.15.94:0/558803027 shutdown_connections 2023-09-30T00:17:35.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.802+0000 7f20474ef700 1 --2- 172.21.15.94:0/558803027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f204013fd70 0x7f20400a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.802+0000 7f20474ef700 1 --2- 172.21.15.94:0/558803027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2040070c50 0x7f204013a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.802+0000 7f20474ef700 1 --2- 172.21.15.94:0/558803027 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f20400b7160 0x7f20400a5fe0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:35.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.802+0000 7f20474ef700 1 -- 172.21.15.94:0/558803027 >> 172.21.15.94:0/558803027 conn(0x7f204006a090 msgr2=0x7f2040141010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:35.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.802+0000 7f20474ef700 1 -- 172.21.15.94:0/558803027 shutdown_connections 2023-09-30T00:17:35.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:35.802+0000 7f20474ef700 1 -- 172.21.15.94:0/558803027 wait complete. 2023-09-30T00:17:35.816 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T00:17:35.816 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T00:17:35.859 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:35.858+0000 7f71f7231180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:17:35.864 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:35.863+0000 7f67d01f4180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:17:35.953 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:35.952+0000 7fce027ab180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:17:36.053 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:36.052+0000 7f71f7231180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:17:36.057 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:36.056+0000 7f67d01f4180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:17:36.124 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:36.123+0000 7fce027ab180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:17:36.189 INFO:tasks.ceph.osd.0.smithi094.stderr:2023-09-30T00:17:36.187+0000 7fdc11834540 -1 Falling back to public interface 2023-09-30T00:17:36.201 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:36.199+0000 7fce027ab180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:17:36.201 INFO:tasks.ceph.osd.2.smithi121.stderr:2023-09-30T00:17:36.199+0000 7f254ce7f540 -1 Falling back to public interface 2023-09-30T00:17:36.201 INFO:tasks.ceph.osd.3.smithi121.stderr:2023-09-30T00:17:36.200+0000 7f5b66508540 -1 Falling back to public interface 2023-09-30T00:17:36.218 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:17:36.225 INFO:tasks.ceph.osd.1.smithi121.stderr:2023-09-30T00:17:36.224+0000 7fd95dc3a540 -1 Falling back to public interface 2023-09-30T00:17:36.234 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:36.234+0000 7f67d01f4180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:17:36.235 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:36.234+0000 7f71f7231180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:17:36.315 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:36.314+0000 7f67d01f4180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:17:36.316 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:36.314+0000 7f71f7231180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:17:36.328 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.326+0000 7f8486122700 1 Processor -- start 2023-09-30T00:17:36.328 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.326+0000 7f8486122700 1 -- start start 2023-09-30T00:17:36.330 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.327+0000 7f8486122700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84800b23d0 0x7f8480148e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:36.330 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.327+0000 7f8486122700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84800b3650 con 0x7f84800a5970 2023-09-30T00:17:36.330 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.327+0000 7f8486122700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8480140fd0 con 0x7f8480140a70 2023-09-30T00:17:36.331 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.327+0000 7f8486122700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8480141290 con 0x7f84800b23d0 2023-09-30T00:17:36.331 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.327+0000 7f847ffff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84800b23d0 0x7f8480148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:36.331 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.327+0000 7f847f7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f84800a5970 0x7f84800b3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:44062/0 (socket says 172.21.15.94:44062) 2023-09-30T00:17:36.331 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.327+0000 7f847f7fe700 1 -- 172.21.15.94:0/518316121 learned_addr learned my addr 172.21.15.94:0/518316121 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:36.331 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.327+0000 7f847ffff700 1 -- 172.21.15.94:0/518316121 >> v1:172.21.15.121:6789/0 conn(0x7f8480140a70 legacy=0x7f848009fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:36.331 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.327+0000 7f847ffff700 1 -- 172.21.15.94:0/518316121 >> v1:172.21.15.94:6789/0 conn(0x7f84800a5970 legacy=0x7f84800b3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:36.331 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.327+0000 7f847e7fc700 1 -- 172.21.15.94:0/518316121 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892323924 0 0) 0x7f84800b3650 con 0x7f84800a5970 2023-09-30T00:17:36.331 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.327+0000 7f847ffff700 1 -- 172.21.15.94:0/518316121 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84800a5d40 con 0x7f84800b23d0 2023-09-30T00:17:36.331 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.328+0000 7f847ffff700 1 --2- 172.21.15.94:0/518316121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84800b23d0 0x7f8480148e20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f847800d0d0 tx=0x7f847800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c813e616848163b9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:36.332 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.328+0000 7f847e7fc700 1 -- 172.21.15.94:0/518316121 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478010070 con 0x7f84800b23d0 2023-09-30T00:17:36.332 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.328+0000 7f847e7fc700 1 -- 172.21.15.94:0/518316121 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847800de80 con 0x7f84800b23d0 2023-09-30T00:17:36.332 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.328+0000 7f847e7fc700 1 -- 172.21.15.94:0/518316121 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847800b320 con 0x7f84800b23d0 2023-09-30T00:17:36.332 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.328+0000 7f8486122700 1 -- 172.21.15.94:0/518316121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84800b23d0 msgr2=0x7f8480148e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:36.332 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.328+0000 7f8486122700 1 --2- 172.21.15.94:0/518316121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84800b23d0 0x7f8480148e20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f847800d0d0 tx=0x7f847800b990 comp rx=0 tx=0).stop 2023-09-30T00:17:36.332 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.328+0000 7f8486122700 1 -- 172.21.15.94:0/518316121 shutdown_connections 2023-09-30T00:17:36.332 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.328+0000 7f8486122700 1 --2- 172.21.15.94:0/518316121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84800b23d0 0x7f8480148e20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:36.332 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.328+0000 7f8486122700 1 -- 172.21.15.94:0/518316121 >> 172.21.15.94:0/518316121 conn(0x7f848006bae0 msgr2=0x7f84800a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:36.332 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.328+0000 7f8486122700 1 -- 172.21.15.94:0/518316121 shutdown_connections 2023-09-30T00:17:36.332 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.328+0000 7f8486122700 1 -- 172.21.15.94:0/518316121 wait complete. 2023-09-30T00:17:36.335 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.329+0000 7f8486122700 1 Processor -- start 2023-09-30T00:17:36.335 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.329+0000 7f8486122700 1 -- start start 2023-09-30T00:17:36.335 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f8486122700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84800a5970 0x7f848013b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:36.335 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f8486122700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84800b23d0 0x7f84800a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:36.335 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f8486122700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8480140a70 0x7f848014f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:36.335 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f8486122700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84800a38b0 con 0x7f84800a5970 2023-09-30T00:17:36.335 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f8486122700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84800a9b20 con 0x7f8480140a70 2023-09-30T00:17:36.336 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f8486122700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84800a9fa0 con 0x7f84800b23d0 2023-09-30T00:17:36.336 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f847f7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84800a5970 0x7f848013b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:36.336 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f847f7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84800a5970 0x7f848013b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46492/0 (socket says 172.21.15.94:46492) 2023-09-30T00:17:36.336 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f847effd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84800b23d0 0x7f84800a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:36.336 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f847f7fe700 1 -- 172.21.15.94:0/2353371892 learned_addr learned my addr 172.21.15.94:0/2353371892 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:36.336 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f847ffff700 1 --2- 172.21.15.94:0/2353371892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8480140a70 0x7f848014f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:36.336 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f847f7fe700 1 -- 172.21.15.94:0/2353371892 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84800b23d0 msgr2=0x7f84800a95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:36.336 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f847f7fe700 1 --2- 172.21.15.94:0/2353371892 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84800b23d0 0x7f84800a95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:36.336 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f847f7fe700 1 -- 172.21.15.94:0/2353371892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8480140a70 msgr2=0x7f848014f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:36.336 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f847f7fe700 1 --2- 172.21.15.94:0/2353371892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8480140a70 0x7f848014f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:36.337 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f847f7fe700 1 -- 172.21.15.94:0/2353371892 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847800bc30 con 0x7f84800a5970 2023-09-30T00:17:36.337 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.330+0000 7f847f7fe700 1 --2- 172.21.15.94:0/2353371892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84800a5970 0x7f848013b560 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f846800c810 tx=0x7f8468007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:36.337 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.331+0000 7f84777fe700 1 -- 172.21.15.94:0/2353371892 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8468005c00 con 0x7f84800a5970 2023-09-30T00:17:36.337 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.331+0000 7f84777fe700 1 -- 172.21.15.94:0/2353371892 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8468005d60 con 0x7f84800a5970 2023-09-30T00:17:36.337 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.331+0000 7f84777fe700 1 -- 172.21.15.94:0/2353371892 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8468007320 con 0x7f84800a5970 2023-09-30T00:17:36.337 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.331+0000 7f8486122700 1 -- 172.21.15.94:0/2353371892 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848014ffe0 con 0x7f84800a5970 2023-09-30T00:17:36.337 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.331+0000 7f8486122700 1 -- 172.21.15.94:0/2353371892 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848014fac0 con 0x7f84800a5970 2023-09-30T00:17:36.337 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.331+0000 7f84777fe700 1 -- 172.21.15.94:0/2353371892 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f846801c6c0 con 0x7f84800a5970 2023-09-30T00:17:36.337 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.331+0000 7f84777fe700 20 mgrc handle_mgr_map mgrmap(e 1) v1 2023-09-30T00:17:36.337 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.331+0000 7f84777fe700 4 mgrc handle_mgr_map Got map version 1 2023-09-30T00:17:36.338 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.331+0000 7f84777fe700 4 mgrc handle_mgr_map Active mgr is now 2023-09-30T00:17:36.338 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.331+0000 7f84777fe700 4 mgrc reconnect No active mgr available yet 2023-09-30T00:17:36.338 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.331+0000 7f84777fe700 1 -- 172.21.15.94:0/2353371892 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f8468007750 con 0x7f84800a5970 2023-09-30T00:17:36.338 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.332+0000 7f8486122700 1 -- 172.21.15.94:0/2353371892 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848006fbd0 con 0x7f84800a5970 2023-09-30T00:17:36.338 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.334+0000 7f84777fe700 1 -- 172.21.15.94:0/2353371892 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8468017070 con 0x7f84800a5970 2023-09-30T00:17:36.349 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:36.348+0000 7fce027ab180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:17:36.415 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.414+0000 7f8486122700 1 -- 172.21.15.94:0/2353371892 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84800b2870 con 0x7f84800a5970 2023-09-30T00:17:36.416 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.414+0000 7f84777fe700 1 -- 172.21.15.94:0/2353371892 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) v1 ==== 74+0+4108 (secure 0 0 0) 0x7f846801cb50 con 0x7f84800a5970 2023-09-30T00:17:36.416 INFO:teuthology.misc.health.smithi094.stdout: 2023-09-30T00:17:36.416 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":6,"fsid":"3c891173-587e-4c6f-9c6a-04ec750eedad","created":"2023-09-30T00:17:34.347592+0000","modified":"2023-09-30T00:17:35.583427+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:17:35.583427+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"c508b402-f8ce-4461-823b-9e35fcd8d40f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e96ecdf0-bb38-4eb1-be99-54f813c12888","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"68700ae4-764c-4dff-9cf2-5f33132cd0d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"54f17ec8-2561-42e8-bbd7-262130ffdac0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:17:36.417 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.415+0000 7f8486122700 10 mgrc shutdown 2023-09-30T00:17:36.417 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.415+0000 7f8486122700 1 -- 172.21.15.94:0/2353371892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84800a5970 msgr2=0x7f848013b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:36.417 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.415+0000 7f8486122700 1 --2- 172.21.15.94:0/2353371892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84800a5970 0x7f848013b560 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f846800c810 tx=0x7f8468007d90 comp rx=0 tx=0).stop 2023-09-30T00:17:36.417 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.415+0000 7f8486122700 1 -- 172.21.15.94:0/2353371892 shutdown_connections 2023-09-30T00:17:36.417 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.415+0000 7f8486122700 1 --2- 172.21.15.94:0/2353371892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8480140a70 0x7f848014f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:36.417 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.415+0000 7f8486122700 1 --2- 172.21.15.94:0/2353371892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84800a5970 0x7f848013b560 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:36.418 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.415+0000 7f8486122700 1 --2- 172.21.15.94:0/2353371892 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84800b23d0 0x7f84800a95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:36.418 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.415+0000 7f8486122700 1 -- 172.21.15.94:0/2353371892 >> 172.21.15.94:0/2353371892 conn(0x7f848006bae0 msgr2=0x7f848006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:36.418 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.415+0000 7f8486122700 1 -- 172.21.15.94:0/2353371892 shutdown_connections 2023-09-30T00:17:36.418 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:36.415+0000 7f8486122700 1 -- 172.21.15.94:0/2353371892 wait complete. 2023-09-30T00:17:36.427 DEBUG:teuthology.misc:0 of 4 OSDs are up 2023-09-30T00:17:36.450 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:36.449+0000 7fce027ab180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:17:36.460 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:36.459+0000 7f67d01f4180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:17:36.460 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:36.459+0000 7f71f7231180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:17:36.559 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:36.558+0000 7f71f7231180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:17:36.560 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:36.559+0000 7f67d01f4180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:17:36.983 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:36.982+0000 7fce027ab180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:17:37.061 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:37.060+0000 7fce027ab180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:17:37.085 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:37.084+0000 7f67d01f4180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:17:37.086 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:37.084+0000 7f71f7231180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:17:37.138 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:37.136+0000 7fce027ab180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:17:37.166 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:37.165+0000 7f71f7231180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:17:37.166 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:37.165+0000 7f67d01f4180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:17:37.247 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:37.246+0000 7f67d01f4180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:17:37.247 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:37.246+0000 7f71f7231180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:17:37.718 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:37.717+0000 7fce027ab180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T00:17:37.854 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:37.853+0000 7fce027ab180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:17:37.871 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:37.870+0000 7f67d01f4180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T00:17:37.872 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:37.871+0000 7f71f7231180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T00:17:37.933 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:37.932+0000 7fce027ab180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:17:38.014 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:38.013+0000 7f67d01f4180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:17:38.016 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:38.015+0000 7f71f7231180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:17:38.113 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:38.111+0000 7f67d01f4180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:17:38.113 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:38.111+0000 7f71f7231180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:17:38.181 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:38.180+0000 7fce027ab180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T00:17:38.269 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:38.267+0000 7fce027ab180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:17:38.375 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:38.374+0000 7f67d01f4180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T00:17:38.375 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:38.374+0000 7f71f7231180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T00:17:38.456 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:38.455+0000 7f71f7231180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:17:38.457 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:38.455+0000 7f67d01f4180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:17:38.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:17:38.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:17:38.948 INFO:tasks.ceph.osd.0.smithi094.stderr:2023-09-30T00:17:38.947+0000 7fdc11834540 -1 osd.0 0 log_to_monitors true 2023-09-30T00:17:39.091 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:39.090+0000 7fce027ab180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T00:17:39.166 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:39.165+0000 7fce027ab180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:17:39.240 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:39.239+0000 7fce027ab180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:17:39.364 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:39.364+0000 7f71f7231180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T00:17:39.372 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:39.371+0000 7fce027ab180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:17:39.386 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:39.385+0000 7f67d01f4180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T00:17:39.443 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:39.442+0000 7f71f7231180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:17:39.456 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:39.454+0000 7fce027ab180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:17:39.464 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:39.462+0000 7f67d01f4180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:17:39.528 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:39.527+0000 7f71f7231180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:17:39.539 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:39.538+0000 7f67d01f4180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:17:39.606 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:39.604+0000 7fce027ab180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:17:39.668 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:39.667+0000 7f71f7231180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:17:39.670 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:39.669+0000 7f67d01f4180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:17:39.681 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:39.680+0000 7fce027ab180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:17:39.701 INFO:tasks.ceph.osd.3.smithi121.stderr:2023-09-30T00:17:39.700+0000 7f5b66508540 -1 osd.3 0 log_to_monitors true 2023-09-30T00:17:39.737 INFO:tasks.ceph.osd.2.smithi121.stderr:2023-09-30T00:17:39.736+0000 7f254ce7f540 -1 osd.2 0 log_to_monitors true 2023-09-30T00:17:39.756 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:39.754+0000 7fce027ab180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:17:39.758 INFO:tasks.ceph.osd.1.smithi121.stderr:2023-09-30T00:17:39.757+0000 7fd95dc3a540 -1 osd.1 0 log_to_monitors true 2023-09-30T00:17:39.763 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:39.762+0000 7f67d01f4180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:17:39.767 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:39.766+0000 7f71f7231180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:17:39.830 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:39.829+0000 7fce027ab180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:17:39.873 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:39.872+0000 7fcd92aa0700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:17:39.873 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:39.872+0000 7fcd92aa0700 -1 pg_autoscaler.serve: 2023-09-30T00:17:39.874 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:39.872+0000 7fcd92aa0700 -1 Traceback (most recent call last): 2023-09-30T00:17:39.874 INFO:tasks.ceph.mgr.x.smithi094.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T00:17:39.874 INFO:tasks.ceph.mgr.x.smithi094.stderr: self._maybe_adjust() 2023-09-30T00:17:39.875 INFO:tasks.ceph.mgr.x.smithi094.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T00:17:39.875 INFO:tasks.ceph.mgr.x.smithi094.stderr: if self.noautoscale: 2023-09-30T00:17:39.875 INFO:tasks.ceph.mgr.x.smithi094.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:17:39.875 INFO:tasks.ceph.mgr.x.smithi094.stderr: 2023-09-30T00:17:39.925 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:39.924+0000 7f67d01f4180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:17:39.926 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:39.924+0000 7f71f7231180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:17:40.006 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:40.005+0000 7f71f7231180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:17:40.006 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:40.005+0000 7f67d01f4180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:17:40.085 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:40.084+0000 7f71f7231180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:17:40.085 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:40.084+0000 7f67d01f4180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:17:40.165 INFO:tasks.ceph.mgr.y.smithi121.stderr:2023-09-30T00:17:40.164+0000 7f67d01f4180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:17:40.166 INFO:tasks.ceph.mgr.z.smithi121.stderr:2023-09-30T00:17:40.164+0000 7f71f7231180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:17:40.669 INFO:tasks.ceph.osd.0.smithi094.stderr:2023-09-30T00:17:40.668+0000 7fdc08455700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T00:17:40.671 INFO:tasks.ceph.osd.0.smithi094.stderr:2023-09-30T00:17:40.670+0000 7fdc0124c700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:17:40.857 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:40.855+0000 7fcdbbbb1700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T00:17:40.857 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-09-30T00:17:40.855+0000 7fcdbbbb1700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T00:17:41.857 INFO:tasks.ceph.mon.a.smithi094.stderr:2023-09-30T00:17:41.856+0000 7fbf70519700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T00:17:42.661 INFO:tasks.ceph.osd.1.smithi121.stderr:2023-09-30T00:17:42.659+0000 7fd95485b700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T00:17:42.663 INFO:tasks.ceph.osd.1.smithi121.stderr:2023-09-30T00:17:42.663+0000 7fd94d652700 -1 osd.1 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:17:42.683 INFO:tasks.ceph.osd.2.smithi121.stderr:2023-09-30T00:17:42.682+0000 7f2543aa0700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T00:17:42.685 INFO:tasks.ceph.osd.2.smithi121.stderr:2023-09-30T00:17:42.684+0000 7f253c897700 -1 osd.2 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:17:42.688 INFO:tasks.ceph.osd.3.smithi121.stderr:2023-09-30T00:17:42.687+0000 7f5b5e92c700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T00:17:42.693 INFO:tasks.ceph.osd.3.smithi121.stderr:2023-09-30T00:17:42.690+0000 7f5b55f20700 -1 osd.3 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:17:42.831 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:17:42.950 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e58b36700 1 Processor -- start 2023-09-30T00:17:42.950 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e58b36700 1 -- start start 2023-09-30T00:17:42.952 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e58b36700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0e540a72c0 0x7f0e5409e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:42.952 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e58b36700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5406b330 con 0x7f0e5406af60 2023-09-30T00:17:42.953 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e58b36700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5409eb90 con 0x7f0e540a51c0 2023-09-30T00:17:42.953 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e58b36700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e540a5e50 con 0x7f0e540a72c0 2023-09-30T00:17:42.953 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e4bfff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0e540a72c0 0x7f0e5409e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:42.953 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e5259c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0e5406af60 0x7f0e5406e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:44068/0 (socket says 172.21.15.94:44068) 2023-09-30T00:17:42.953 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e4bfff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0e540a72c0 0x7f0e5409e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46564/0 (socket says 172.21.15.94:46564) 2023-09-30T00:17:42.953 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e5259c700 1 -- 172.21.15.94:0/3285352820 learned_addr learned my addr 172.21.15.94:0/3285352820 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:42.953 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e4bfff700 1 -- 172.21.15.94:0/3285352820 >> v1:172.21.15.121:6789/0 conn(0x7f0e540a51c0 legacy=0x7f0e540a5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:42.953 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e4bfff700 1 -- 172.21.15.94:0/3285352820 >> v1:172.21.15.94:6789/0 conn(0x7f0e5406af60 legacy=0x7f0e5406e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:42.953 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e4bfff700 1 -- 172.21.15.94:0/3285352820 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5406da90 con 0x7f0e540a72c0 2023-09-30T00:17:42.953 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e51d9b700 1 -- 172.21.15.94:0/3285352820 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298654102 0 0) 0x7f0e5406b330 con 0x7f0e5406af60 2023-09-30T00:17:42.954 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e4bfff700 1 --2- 172.21.15.94:0/3285352820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e540a72c0 0x7f0e5409e550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c0098e0 tx=0x7f0e3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91a99419e1fde24 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:42.954 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e51d9b700 1 -- 172.21.15.94:0/3285352820 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c009e80 con 0x7f0e540a72c0 2023-09-30T00:17:42.954 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e51d9b700 1 -- 172.21.15.94:0/3285352820 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3c00b820 con 0x7f0e540a72c0 2023-09-30T00:17:42.954 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.950+0000 7f0e51d9b700 1 -- 172.21.15.94:0/3285352820 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c00bba0 con 0x7f0e540a72c0 2023-09-30T00:17:42.954 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.951+0000 7f0e58b36700 1 -- 172.21.15.94:0/3285352820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e540a72c0 msgr2=0x7f0e5409e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:42.954 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.951+0000 7f0e58b36700 1 --2- 172.21.15.94:0/3285352820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e540a72c0 0x7f0e5409e550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c0098e0 tx=0x7f0e3c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:42.954 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.951+0000 7f0e58b36700 1 -- 172.21.15.94:0/3285352820 shutdown_connections 2023-09-30T00:17:42.954 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.951+0000 7f0e58b36700 1 --2- 172.21.15.94:0/3285352820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e540a72c0 0x7f0e5409e550 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:42.954 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.951+0000 7f0e58b36700 1 -- 172.21.15.94:0/3285352820 >> 172.21.15.94:0/3285352820 conn(0x7f0e54067620 msgr2=0x7f0e54067a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:42.954 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.951+0000 7f0e58b36700 1 -- 172.21.15.94:0/3285352820 shutdown_connections 2023-09-30T00:17:42.954 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.951+0000 7f0e58b36700 1 -- 172.21.15.94:0/3285352820 wait complete. 2023-09-30T00:17:42.957 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e58b36700 1 Processor -- start 2023-09-30T00:17:42.957 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e58b36700 1 -- start start 2023-09-30T00:17:42.957 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e58b36700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e5406af60 0x7f0e5413dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:42.957 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e58b36700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e540a51c0 0x7f0e54143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:42.957 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e58b36700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e5413cbe0 0x7f0e5413d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:42.957 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e58b36700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5413c920 con 0x7f0e540a51c0 2023-09-30T00:17:42.957 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e58b36700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e54135320 con 0x7f0e5406af60 2023-09-30T00:17:42.958 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e58b36700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5413a1d0 con 0x7f0e5413cbe0 2023-09-30T00:17:42.958 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e4bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e540a51c0 0x7f0e54143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:42.958 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e4bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e540a51c0 0x7f0e54143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46570/0 (socket says 172.21.15.94:46570) 2023-09-30T00:17:42.958 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e4bfff700 1 -- 172.21.15.94:0/3798474362 learned_addr learned my addr 172.21.15.94:0/3798474362 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:42.958 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e5259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e5406af60 0x7f0e5413dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:42.958 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.952+0000 7f0e52d9d700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e5413cbe0 0x7f0e5413d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:42.958 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.953+0000 7f0e4bfff700 1 -- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e5413cbe0 msgr2=0x7f0e5413d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:42.958 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.953+0000 7f0e4bfff700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e5413cbe0 0x7f0e5413d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:42.958 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.953+0000 7f0e4bfff700 1 -- 172.21.15.94:0/3798474362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e5406af60 msgr2=0x7f0e5413dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:42.958 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.953+0000 7f0e4bfff700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e5406af60 0x7f0e5413dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.953+0000 7f0e4bfff700 1 -- 172.21.15.94:0/3798474362 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c009420 con 0x7f0e540a51c0 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.953+0000 7f0e4bfff700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e540a51c0 0x7f0e54143600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c0030c0 tx=0x7f0e3c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.953+0000 7f0e4a7fc700 1 -- 172.21.15.94:0/3798474362 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c00b990 con 0x7f0e540a51c0 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.953+0000 7f0e4a7fc700 1 -- 172.21.15.94:0/3798474362 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3c020070 con 0x7f0e540a51c0 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.953+0000 7f0e58b36700 1 -- 172.21.15.94:0/3798474362 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5413e120 con 0x7f0e540a51c0 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.953+0000 7f0e4a7fc700 1 -- 172.21.15.94:0/3798474362 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c01b6a0 con 0x7f0e540a51c0 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.953+0000 7f0e58b36700 1 -- 172.21.15.94:0/3798474362 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5413b8f0 con 0x7f0e540a51c0 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.954+0000 7f0e58b36700 1 -- 172.21.15.94:0/3798474362 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e54143320 con 0x7f0e540a51c0 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.955+0000 7f0e4a7fc700 1 -- 172.21.15.94:0/3798474362 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) v1 ==== 126883+0+0 (secure 0 0 0) 0x7f0e3c022030 con 0x7f0e540a51c0 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.955+0000 7f0e4a7fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.955+0000 7f0e4a7fc700 4 mgrc handle_mgr_map Got map version 4 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.955+0000 7f0e4a7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:42.959 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.955+0000 7f0e4a7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:42.960 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.955+0000 7f0e4a7fc700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f0e34098720 0x7f0e3409abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:42.960 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.956+0000 7f0e5259c700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f0e34098720 0x7f0e3409abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:42.960 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.956+0000 7f0e4a7fc700 1 -- 172.21.15.94:0/3798474362 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 2739+0+0 (secure 0 0 0) 0x7f0e3c0c1aa0 con 0x7f0e540a51c0 2023-09-30T00:17:42.960 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.956+0000 7f0e5259c700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f0e34098720 0x7f0e3409abd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0e44007fd0 tx=0x7f0e44007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:42.961 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:42.960+0000 7f0e4a7fc700 1 -- 172.21.15.94:0/3798474362 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0e54143320 con 0x7f0e540a51c0 2023-09-30T00:17:43.145 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.144+0000 7f0e58b36700 1 -- 172.21.15.94:0/3798474362 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e54143320 con 0x7f0e540a51c0 2023-09-30T00:17:43.146 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.145+0000 7f0e4a7fc700 1 -- 172.21.15.94:0/3798474362 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+5846 (secure 0 0 0) 0x7f0e54143320 con 0x7f0e540a51c0 2023-09-30T00:17:43.146 INFO:teuthology.misc.health.smithi094.stdout: 2023-09-30T00:17:43.146 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"3c891173-587e-4c6f-9c6a-04ec750eedad","created":"2023-09-30T00:17:34.347592+0000","modified":"2023-09-30T00:17:42.879328+0000","last_up_change":"2023-09-30T00:17:42.879328+0000","last_in_change":"2023-09-30T00:17:35.583427+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"c508b402-f8ce-4461-823b-9e35fcd8d40f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":84364},{"type":"v1","addr":"172.21.15.94:6801","nonce":84364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":84364},{"type":"v1","addr":"172.21.15.94:6803","nonce":84364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":84364},{"type":"v1","addr":"172.21.15.94:6807","nonce":84364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":84364},{"type":"v1","addr":"172.21.15.94:6805","nonce":84364}]},"public_addr":"172.21.15.94:6801/84364","cluster_addr":"172.21.15.94:6803/84364","heartbeat_back_addr":"172.21.15.94:6807/84364","heartbeat_front_addr":"172.21.15.94:6805/84364","state":["exists","up"]},{"osd":1,"uuid":"e96ecdf0-bb38-4eb1-be99-54f813c12888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":86850},{"type":"v1","addr":"172.21.15.121:6817","nonce":86850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":86850},{"type":"v1","addr":"172.21.15.121:6819","nonce":86850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":86850},{"type":"v1","addr":"172.21.15.121:6823","nonce":86850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":86850},{"type":"v1","addr":"172.21.15.121:6821","nonce":86850}]},"public_addr":"172.21.15.121:6817/86850","cluster_addr":"172.21.15.121:6819/86850","heartbeat_back_addr":"172.21.15.121:6823/86850","heartbeat_front_addr":"172.21.15.121:6821/86850","state":["exists","up"]},{"osd":2,"uuid":"68700ae4-764c-4dff-9cf2-5f33132cd0d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":86820},{"type":"v1","addr":"172.21.15.121:6801","nonce":86820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":86820},{"type":"v1","addr":"172.21.15.121:6803","nonce":86820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":86820},{"type":"v1","addr":"172.21.15.121:6810","nonce":86820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":86820},{"type":"v1","addr":"172.21.15.121:6806","nonce":86820}]},"public_addr":"172.21.15.121:6801/86820","cluster_addr":"172.21.15.121:6803/86820","heartbeat_back_addr":"172.21.15.121:6810/86820","heartbeat_front_addr":"172.21.15.121:6806/86820","state":["exists","up"]},{"osd":3,"uuid":"54f17ec8-2561-42e8-bbd7-262130ffdac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":86821},{"type":"v1","addr":"172.21.15.121:6807","nonce":86821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":86821},{"type":"v1","addr":"172.21.15.121:6811","nonce":86821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":86821},{"type":"v1","addr":"172.21.15.121:6815","nonce":86821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":86821},{"type":"v1","addr":"172.21.15.121:6813","nonce":86821}]},"public_addr":"172.21.15.121:6807/86821","cluster_addr":"172.21.15.121:6811/86821","heartbeat_back_addr":"172.21.15.121:6815/86821","heartbeat_front_addr":"172.21.15.121:6813/86821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:39.951865+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:17:43.148 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 10 mgrc shutdown 2023-09-30T00:17:43.148 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 1 -- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f0e34098720 msgr2=0x7f0e3409abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:43.148 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f0e34098720 0x7f0e3409abd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0e44007fd0 tx=0x7f0e44007d30 comp rx=0 tx=0).stop 2023-09-30T00:17:43.148 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 1 -- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e540a51c0 msgr2=0x7f0e54143600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:43.148 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e540a51c0 0x7f0e54143600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c0030c0 tx=0x7f0e3c016000 comp rx=0 tx=0).stop 2023-09-30T00:17:43.148 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 1 -- 172.21.15.94:0/3798474362 shutdown_connections 2023-09-30T00:17:43.148 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f0e34098720 0x7f0e3409abd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:43.148 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e5413cbe0 0x7f0e5413d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:43.148 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e5406af60 0x7f0e5413dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:43.148 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 1 --2- 172.21.15.94:0/3798474362 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e540a51c0 0x7f0e54143600 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:43.149 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 1 -- 172.21.15.94:0/3798474362 >> 172.21.15.94:0/3798474362 conn(0x7f0e54067620 msgr2=0x7f0e540a3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:43.149 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 1 -- 172.21.15.94:0/3798474362 shutdown_connections 2023-09-30T00:17:43.149 INFO:teuthology.misc.health.smithi094.stderr:2023-09-30T00:17:43.147+0000 7f0e58b36700 1 -- 172.21.15.94:0/3798474362 wait complete. 2023-09-30T00:17:43.158 DEBUG:teuthology.misc:4 of 4 OSDs are up 2023-09-30T00:17:43.158 INFO:tasks.ceph:Creating RBD pool 2023-09-30T00:17:43.159 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T00:17:43.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.273+0000 7fbff4504700 1 Processor -- start 2023-09-30T00:17:43.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.273+0000 7fbff4504700 1 -- start start 2023-09-30T00:17:43.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.273+0000 7fbff4504700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbfec0a6ad0 0x7fbfec09e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:43.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.273+0000 7fbff4504700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec141c20 0x7fbfec142060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:43.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.273+0000 7fbff4504700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfec0a6ea0 con 0x7fbfec0a7be0 2023-09-30T00:17:43.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.273+0000 7fbff4504700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfec1425a0 con 0x7fbfec0a6ad0 2023-09-30T00:17:43.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.273+0000 7fbff4504700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec09e990 con 0x7fbfec141c20 2023-09-30T00:17:43.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff2aa1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec141c20 0x7fbfec142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:43.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff1a9f700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbfec0a7be0 0x7fbfec134830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:44080/0 (socket says 172.21.15.94:44080) 2023-09-30T00:17:43.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff1a9f700 1 -- 172.21.15.94:0/190331465 learned_addr learned my addr 172.21.15.94:0/190331465 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:43.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff22a0700 1 --2- 172.21.15.94:0/190331465 >> v2:172.21.15.121:3300/0 conn(0x7fbfec0a6ad0 0x7fbfec09e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:43.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff2aa1700 1 -- 172.21.15.94:0/190331465 >> v2:172.21.15.121:3300/0 conn(0x7fbfec0a6ad0 msgr2=0x7fbfec09e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:43.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff129e700 1 -- 172.21.15.94:0/190331465 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963131994 0 0) 0x7fbfec0a6ea0 con 0x7fbfec0a7be0 2023-09-30T00:17:43.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff2aa1700 1 --2- 172.21.15.94:0/190331465 >> v2:172.21.15.121:3300/0 conn(0x7fbfec0a6ad0 0x7fbfec09e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:43.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff2aa1700 1 -- 172.21.15.94:0/190331465 >> v1:172.21.15.94:6789/0 conn(0x7fbfec0a7be0 legacy=0x7fbfec134830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:43.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff2aa1700 1 -- 172.21.15.94:0/190331465 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec134f40 con 0x7fbfec141c20 2023-09-30T00:17:43.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff2aa1700 1 --2- 172.21.15.94:0/190331465 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec141c20 0x7fbfec142060 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40098e0 tx=0x7fbfe40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7aa0bce5b3582e6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:43.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff129e700 1 -- 172.21.15.94:0/190331465 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4009e80 con 0x7fbfec141c20 2023-09-30T00:17:43.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff129e700 1 -- 172.21.15.94:0/190331465 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe400b860 con 0x7fbfec141c20 2023-09-30T00:17:43.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.274+0000 7fbff129e700 1 -- 172.21.15.94:0/190331465 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe400bbe0 con 0x7fbfec141c20 2023-09-30T00:17:43.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.275+0000 7fbff4504700 1 -- 172.21.15.94:0/190331465 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec141c20 msgr2=0x7fbfec142060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.275+0000 7fbff4504700 1 --2- 172.21.15.94:0/190331465 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec141c20 0x7fbfec142060 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40098e0 tx=0x7fbfe40091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.275+0000 7fbff4504700 1 -- 172.21.15.94:0/190331465 shutdown_connections 2023-09-30T00:17:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.275+0000 7fbff4504700 1 --2- 172.21.15.94:0/190331465 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec141c20 0x7fbfec142060 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.275+0000 7fbff4504700 1 --2- 172.21.15.94:0/190331465 >> v2:172.21.15.121:3300/0 conn(0x7fbfec0a6ad0 0x7fbfec09e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.275+0000 7fbff4504700 1 -- 172.21.15.94:0/190331465 >> 172.21.15.94:0/190331465 conn(0x7fbfec06a940 msgr2=0x7fbfec06ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.275+0000 7fbff4504700 1 -- 172.21.15.94:0/190331465 shutdown_connections 2023-09-30T00:17:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.275+0000 7fbff4504700 1 -- 172.21.15.94:0/190331465 wait complete. 2023-09-30T00:17:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff4504700 1 Processor -- start 2023-09-30T00:17:43.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff4504700 1 -- start start 2023-09-30T00:17:43.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff4504700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbfec0a6ad0 0x7fbfec095c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:43.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff4504700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec0a7be0 0x7fbfec097140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:43.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff4504700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfec141c20 0x7fbfec151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:43.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff4504700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec096560 con 0x7fbfec0a6ad0 2023-09-30T00:17:43.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff4504700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec0a2f00 con 0x7fbfec141c20 2023-09-30T00:17:43.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff4504700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec13eca0 con 0x7fbfec0a7be0 2023-09-30T00:17:43.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff1a9f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec0a7be0 0x7fbfec097140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:43.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff22a0700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbfec0a6ad0 0x7fbfec095c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:43.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff22a0700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbfec0a6ad0 0x7fbfec095c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46576/0 (socket says 172.21.15.94:46576) 2023-09-30T00:17:43.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff1a9f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec0a7be0 0x7fbfec097140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44634/0 (socket says 172.21.15.94:44634) 2023-09-30T00:17:43.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff22a0700 1 -- 172.21.15.94:0/385628729 learned_addr learned my addr 172.21.15.94:0/385628729 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:43.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff2aa1700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfec141c20 0x7fbfec151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:43.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff1a9f700 1 -- 172.21.15.94:0/385628729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfec141c20 msgr2=0x7fbfec151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:43.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff1a9f700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfec141c20 0x7fbfec151ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:43.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff1a9f700 1 -- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbfec0a6ad0 msgr2=0x7fbfec095c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:43.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.276+0000 7fbff1a9f700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbfec0a6ad0 0x7fbfec095c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:43.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.277+0000 7fbff1a9f700 1 -- 172.21.15.94:0/385628729 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe4009420 con 0x7fbfec0a7be0 2023-09-30T00:17:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.277+0000 7fbff1a9f700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec0a7be0 0x7fbfec097140 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0011e0 tx=0x7fbfdc007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.277+0000 7fbfe27fc700 1 -- 172.21.15.94:0/385628729 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc005b30 con 0x7fbfec0a7be0 2023-09-30T00:17:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.277+0000 7fbfe27fc700 1 -- 172.21.15.94:0/385628729 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc005c90 con 0x7fbfec0a7be0 2023-09-30T00:17:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.277+0000 7fbfe27fc700 1 -- 172.21.15.94:0/385628729 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc007300 con 0x7fbfec0a7be0 2023-09-30T00:17:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.277+0000 7fbff4504700 1 -- 172.21.15.94:0/385628729 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfec13ef60 con 0x7fbfec0a7be0 2023-09-30T00:17:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.277+0000 7fbff4504700 1 -- 172.21.15.94:0/385628729 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfec0a0b30 con 0x7fbfec0a7be0 2023-09-30T00:17:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.278+0000 7fbff4504700 1 -- 172.21.15.94:0/385628729 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfec096e60 con 0x7fbfec0a7be0 2023-09-30T00:17:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.279+0000 7fbfe27fc700 1 -- 172.21.15.94:0/385628729 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 4) v1 ==== 126883+0+0 (secure 0 0 0) 0x7fbfdc01b3b0 con 0x7fbfec0a7be0 2023-09-30T00:17:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.279+0000 7fbfe27fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2023-09-30T00:17:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.279+0000 7fbfe27fc700 4 mgrc handle_mgr_map Got map version 4 2023-09-30T00:17:43.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.279+0000 7fbfe27fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:43.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.279+0000 7fbfe27fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:43.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.279+0000 7fbfe27fc700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fbfd00986f0 0x7fbfd009aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:43.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.279+0000 7fbff22a0700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fbfd00986f0 0x7fbfd009aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:43.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.279+0000 7fbfe27fc700 1 -- 172.21.15.94:0/385628729 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 2739+0+0 (secure 0 0 0) 0x7fbfdc0bf170 con 0x7fbfec0a7be0 2023-09-30T00:17:43.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.279+0000 7fbff22a0700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fbfd00986f0 0x7fbfd009aba0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8007700 tx=0x7fbfe8009920 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:43.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.283+0000 7fbfe27fc700 1 -- 172.21.15.94:0/385628729 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbfec096e60 con 0x7fbfec0a7be0 2023-09-30T00:17:43.464 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:43.463+0000 7fbff4504700 1 -- 172.21.15.94:0/385628729 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fbfec096e60 con 0x7fbfec0a7be0 2023-09-30T00:17:44.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.190+0000 7fbfe27fc700 1 -- 172.21.15.94:0/385628729 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v11) v1 ==== 109+0+0 (secure 0 0 0) 0x7fbfec096e60 con 0x7fbfec0a7be0 2023-09-30T00:17:44.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 10 mgrc shutdown 2023-09-30T00:17:44.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 1 -- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fbfd00986f0 msgr2=0x7fbfd009aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:44.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fbfd00986f0 0x7fbfd009aba0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8007700 tx=0x7fbfe8009920 comp rx=0 tx=0).stop 2023-09-30T00:17:44.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 1 -- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec0a7be0 msgr2=0x7fbfec097140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:44.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec0a7be0 0x7fbfec097140 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0011e0 tx=0x7fbfdc007d30 comp rx=0 tx=0).stop 2023-09-30T00:17:44.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 1 -- 172.21.15.94:0/385628729 shutdown_connections 2023-09-30T00:17:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fbfd00986f0 0x7fbfd009aba0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfec141c20 0x7fbfec151ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbfec0a6ad0 0x7fbfec095c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 1 --2- 172.21.15.94:0/385628729 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbfec0a7be0 0x7fbfec097140 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 1 -- 172.21.15.94:0/385628729 >> 172.21.15.94:0/385628729 conn(0x7fbfec06a940 msgr2=0x7fbfec0a6000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 1 -- 172.21.15.94:0/385628729 shutdown_connections 2023-09-30T00:17:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.192+0000 7fbff4504700 1 -- 172.21.15.94:0/385628729 wait complete. 2023-09-30T00:17:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'rbd' created 2023-09-30T00:17:44.205 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T00:17:44.321 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.320+0000 7fc9cab4a700 1 Processor -- start 2023-09-30T00:17:44.321 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.321+0000 7fc9cab4a700 1 -- start start 2023-09-30T00:17:44.324 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.321+0000 7fc9cab4a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9c4141d10 0x7fc9c409e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:44.324 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.321+0000 7fc9cab4a700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c40a6ed0 con 0x7fc9c40a56a0 2023-09-30T00:17:44.324 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.321+0000 7fc9cab4a700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c40b5210 con 0x7fc9c40b4e40 2023-09-30T00:17:44.324 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.321+0000 7fc9cab4a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c41421a0 con 0x7fc9c4141d10 2023-09-30T00:17:44.324 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.321+0000 7fc9c3fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9c4141d10 0x7fc9c409e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:44.324 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.321+0000 7fc9c90e7700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc9c40a56a0 0x7fc9c41347b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:44090/0 (socket says 172.21.15.94:44090) 2023-09-30T00:17:44.324 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.321+0000 7fc9c90e7700 1 -- 172.21.15.94:0/1800357113 learned_addr learned my addr 172.21.15.94:0/1800357113 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:44.324 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.321+0000 7fc9c37fe700 1 -- 172.21.15.94:0/1800357113 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280698534 0 0) 0x7fc9bc001110 con 0x7fc9c40a56a0 2023-09-30T00:17:44.324 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9c37fe700 1 -- 172.21.15.94:0/1800357113 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9ac003570 con 0x7fc9c40a56a0 2023-09-30T00:17:44.325 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9c3fff700 1 -- 172.21.15.94:0/1800357113 >> v1:172.21.15.121:6789/0 conn(0x7fc9c40b4e40 legacy=0x7fc9c4142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:44.325 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9c3fff700 1 -- 172.21.15.94:0/1800357113 >> v1:172.21.15.94:6789/0 conn(0x7fc9c40a56a0 legacy=0x7fc9c41347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:44.325 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9c3fff700 1 -- 172.21.15.94:0/1800357113 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c40a5c00 con 0x7fc9c4141d10 2023-09-30T00:17:44.325 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9c3fff700 1 --2- 172.21.15.94:0/1800357113 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9c4141d10 0x7fc9c409e980 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8009910 tx=0x7fc9b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b31c10c9a2b6bba8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:44.325 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9c37fe700 1 -- 172.21.15.94:0/1800357113 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b800b710 con 0x7fc9c4141d10 2023-09-30T00:17:44.325 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9c37fe700 1 -- 172.21.15.94:0/1800357113 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b800b870 con 0x7fc9c4141d10 2023-09-30T00:17:44.325 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9c37fe700 1 -- 172.21.15.94:0/1800357113 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b800bc30 con 0x7fc9c4141d10 2023-09-30T00:17:44.326 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9cab4a700 1 -- 172.21.15.94:0/1800357113 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9c4141d10 msgr2=0x7fc9c409e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:44.326 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9cab4a700 1 --2- 172.21.15.94:0/1800357113 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9c4141d10 0x7fc9c409e980 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8009910 tx=0x7fc9b80091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:44.326 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9cab4a700 1 -- 172.21.15.94:0/1800357113 shutdown_connections 2023-09-30T00:17:44.326 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9cab4a700 1 --2- 172.21.15.94:0/1800357113 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9c4141d10 0x7fc9c409e980 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:44.326 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9cab4a700 1 -- 172.21.15.94:0/1800357113 >> 172.21.15.94:0/1800357113 conn(0x7fc9c406a940 msgr2=0x7fc9c406ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:44.326 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9cab4a700 1 -- 172.21.15.94:0/1800357113 shutdown_connections 2023-09-30T00:17:44.326 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.322+0000 7fc9cab4a700 1 -- 172.21.15.94:0/1800357113 wait complete. 2023-09-30T00:17:44.327 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.323+0000 7fc9cab4a700 1 Processor -- start 2023-09-30T00:17:44.327 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9cab4a700 1 -- start start 2023-09-30T00:17:44.333 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9cab4a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9c40a56a0 0x7fc9c40ab020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:44.333 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9cab4a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9c40b4e40 0x7fc9c4069950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:44.333 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9cab4a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9c4141d10 0x7fc9c40687c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:44.333 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9cab4a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c4069fc0 con 0x7fc9c40a56a0 2023-09-30T00:17:44.333 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9cab4a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c406a280 con 0x7fc9c4141d10 2023-09-30T00:17:44.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9cab4a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40ab7b0 con 0x7fc9c40b4e40 2023-09-30T00:17:44.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9c88e6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9c40a56a0 0x7fc9c40ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:44.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9c3fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9c40b4e40 0x7fc9c4069950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:44.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9c88e6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9c40a56a0 0x7fc9c40ab020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46586/0 (socket says 172.21.15.94:46586) 2023-09-30T00:17:44.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9c88e6700 1 -- 172.21.15.94:0/2507549834 learned_addr learned my addr 172.21.15.94:0/2507549834 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:44.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9c90e7700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9c4141d10 0x7fc9c40687c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:44.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9c3fff700 1 -- 172.21.15.94:0/2507549834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9c4141d10 msgr2=0x7fc9c40687c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:44.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9c3fff700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9c4141d10 0x7fc9c40687c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:44.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9c3fff700 1 -- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9c40a56a0 msgr2=0x7fc9c40ab020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:44.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9c3fff700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9c40a56a0 0x7fc9c40ab020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:44.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9c3fff700 1 -- 172.21.15.94:0/2507549834 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b8009420 con 0x7fc9c40b4e40 2023-09-30T00:17:44.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.324+0000 7fc9c88e6700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9c40a56a0 0x7fc9c40ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:44.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.325+0000 7fc9c3fff700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9c40b4e40 0x7fc9c4069950 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8016000 tx=0x7fc9b8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:44.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.325+0000 7fc9c0ff9700 1 -- 172.21.15.94:0/2507549834 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b801a430 con 0x7fc9c40b4e40 2023-09-30T00:17:44.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.325+0000 7fc9c0ff9700 1 -- 172.21.15.94:0/2507549834 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b801a5d0 con 0x7fc9c40b4e40 2023-09-30T00:17:44.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.325+0000 7fc9cab4a700 1 -- 172.21.15.94:0/2507549834 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c41499d0 con 0x7fc9c40b4e40 2023-09-30T00:17:44.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.325+0000 7fc9c0ff9700 1 -- 172.21.15.94:0/2507549834 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b801a990 con 0x7fc9c40b4e40 2023-09-30T00:17:44.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.325+0000 7fc9cab4a700 1 -- 172.21.15.94:0/2507549834 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c413ef10 con 0x7fc9c40b4e40 2023-09-30T00:17:44.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.326+0000 7fc9cab4a700 1 -- 172.21.15.94:0/2507549834 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c4071630 con 0x7fc9c40b4e40 2023-09-30T00:17:44.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.327+0000 7fc9c0ff9700 1 -- 172.21.15.94:0/2507549834 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 4) v1 ==== 126883+0+0 (secure 0 0 0) 0x7fc9b801ad00 con 0x7fc9c40b4e40 2023-09-30T00:17:44.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.327+0000 7fc9c0ff9700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2023-09-30T00:17:44.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.327+0000 7fc9c0ff9700 4 mgrc handle_mgr_map Got map version 4 2023-09-30T00:17:44.337 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.327+0000 7fc9c0ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:44.337 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.327+0000 7fc9c0ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:44.337 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.327+0000 7fc9c0ff9700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc9b00986f0 0x7fc9b009aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:44.337 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.327+0000 7fc9c88e6700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc9b00986f0 0x7fc9b009aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:44.337 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.328+0000 7fc9c0ff9700 1 -- 172.21.15.94:0/2507549834 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 3081+0+0 (secure 0 0 0) 0x7fc9b80c1bf0 con 0x7fc9c40b4e40 2023-09-30T00:17:44.337 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.328+0000 7fc9c88e6700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc9b00986f0 0x7fc9b009aba0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4002230 tx=0x7fc9b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:44.337 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.332+0000 7fc9c0ff9700 1 -- 172.21.15.94:0/2507549834 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc9b808f8f0 con 0x7fc9c40b4e40 2023-09-30T00:17:44.514 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:44.514+0000 7fc9cab4a700 1 -- 172.21.15.94:0/2507549834 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc9c4149c60 con 0x7fc9c40b4e40 2023-09-30T00:17:45.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.191+0000 7fc9c0ff9700 1 -- 172.21.15.94:0/2507549834 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v12) v1 ==== 173+0+0 (secure 0 0 0) 0x7fc9b8015020 con 0x7fc9c40b4e40 2023-09-30T00:17:45.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 10 mgrc shutdown 2023-09-30T00:17:45.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 1 -- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc9b00986f0 msgr2=0x7fc9b009aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc9b00986f0 0x7fc9b009aba0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4002230 tx=0x7fc9b4009040 comp rx=0 tx=0).stop 2023-09-30T00:17:45.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 1 -- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9c40b4e40 msgr2=0x7fc9c4069950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9c40b4e40 0x7fc9c4069950 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8016000 tx=0x7fc9b8017000 comp rx=0 tx=0).stop 2023-09-30T00:17:45.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 1 -- 172.21.15.94:0/2507549834 shutdown_connections 2023-09-30T00:17:45.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc9b00986f0 0x7fc9b009aba0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9c4141d10 0x7fc9c40687c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9c40a56a0 0x7fc9c40ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 1 --2- 172.21.15.94:0/2507549834 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9c40b4e40 0x7fc9c4069950 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 1 -- 172.21.15.94:0/2507549834 >> 172.21.15.94:0/2507549834 conn(0x7fc9c406a940 msgr2=0x7fc9c40a6610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:45.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 1 -- 172.21.15.94:0/2507549834 shutdown_connections 2023-09-30T00:17:45.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.193+0000 7fc9cab4a700 1 -- 172.21.15.94:0/2507549834 wait complete. 2023-09-30T00:17:45.195 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T00:17:45.207 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T00:17:45.208 INFO:tasks.ceph.mds.a:Restarting daemon 2023-09-30T00:17:45.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-09-30T00:17:45.210 INFO:tasks.ceph.mds.a:Started 2023-09-30T00:17:45.210 INFO:tasks.ceph.mds.c:Restarting daemon 2023-09-30T00:17:45.210 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-09-30T00:17:45.212 INFO:tasks.ceph.mds.c:Started 2023-09-30T00:17:45.213 INFO:tasks.ceph.mds.b:Restarting daemon 2023-09-30T00:17:45.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-09-30T00:17:45.216 INFO:tasks.ceph.mds.b:Started 2023-09-30T00:17:45.216 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-09-30T00:17:45.217 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:17:45.283 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2023-09-30T00:17:45.283 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2023-09-30T00:17:45.296 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2023-09-30T00:17:45.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.338+0000 7f833de28700 1 Processor -- start 2023-09-30T00:17:45.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.338+0000 7f833de28700 1 -- start start 2023-09-30T00:17:45.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.338+0000 7f833de28700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8338070c50 0x7f83380a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.338+0000 7f833de28700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 0x7f83380a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.338+0000 7f833de28700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83380a1310 con 0x7f833813fd70 2023-09-30T00:17:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.338+0000 7f833de28700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8338071020 con 0x7f8338070c50 2023-09-30T00:17:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.338+0000 7f833de28700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380b76a0 con 0x7f83380b7160 2023-09-30T00:17:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.338+0000 7f8336ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 0x7f83380a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.338+0000 7f8337fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f833813fd70 0x7f83380b4dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:32838/0 (socket says 172.21.15.94:32838) 2023-09-30T00:17:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.338+0000 7f8336ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 0x7f83380a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51408/0 (socket says 172.21.15.94:51408) 2023-09-30T00:17:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.338+0000 7f8337fff700 1 -- 172.21.15.94:0/3936046713 learned_addr learned my addr 172.21.15.94:0/3936046713 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.338+0000 7f83377fe700 1 --2- 172.21.15.94:0/3936046713 >> v2:172.21.15.94:3300/0 conn(0x7f8338070c50 0x7f83380a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.339+0000 7f83377fe700 1 -- 172.21.15.94:0/3936046713 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 msgr2=0x7f83380a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.339+0000 7f83377fe700 1 --2- 172.21.15.94:0/3936046713 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 0x7f83380a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.339+0000 7f83377fe700 1 -- 172.21.15.94:0/3936046713 >> v1:172.21.15.94:6789/0 conn(0x7f833813fd70 legacy=0x7f83380b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.339+0000 7f83377fe700 1 -- 172.21.15.94:0/3936046713 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83380b7940 con 0x7f8338070c50 2023-09-30T00:17:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.339+0000 7f83367fc700 1 -- 172.21.15.94:0/3936046713 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686937093 0 0) 0x7f8330001490 con 0x7f833813fd70 2023-09-30T00:17:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.339+0000 7f8336ffd700 1 --2- 172.21.15.94:0/3936046713 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 0x7f83380a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.339+0000 7f83377fe700 1 --2- 172.21.15.94:0/3936046713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8338070c50 0x7f83380a0dd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f83280098e0 tx=0x7f83280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7eae1045b0d22a5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.339+0000 7f83367fc700 1 -- 172.21.15.94:0/3936046713 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832800b6d0 con 0x7f8338070c50 2023-09-30T00:17:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.339+0000 7f83367fc700 1 -- 172.21.15.94:0/3936046713 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832800b830 con 0x7f8338070c50 2023-09-30T00:17:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.339+0000 7f83367fc700 1 -- 172.21.15.94:0/3936046713 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832800bbd0 con 0x7f8338070c50 2023-09-30T00:17:45.343 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.339+0000 7f833de28700 1 -- 172.21.15.94:0/3936046713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8338070c50 msgr2=0x7f83380a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.344 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.339+0000 7f833de28700 1 --2- 172.21.15.94:0/3936046713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8338070c50 0x7f83380a0dd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f83280098e0 tx=0x7f83280091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.344 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.340+0000 7f833de28700 1 -- 172.21.15.94:0/3936046713 shutdown_connections 2023-09-30T00:17:45.344 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.340+0000 7f833de28700 1 --2- 172.21.15.94:0/3936046713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8338070c50 0x7f83380a0dd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.344 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.340+0000 7f833de28700 1 --2- 172.21.15.94:0/3936046713 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 0x7f83380a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.344 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.340+0000 7f833de28700 1 -- 172.21.15.94:0/3936046713 >> 172.21.15.94:0/3936046713 conn(0x7f833806a090 msgr2=0x7f83380702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:45.344 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.340+0000 7f833de28700 1 -- 172.21.15.94:0/3936046713 shutdown_connections 2023-09-30T00:17:45.344 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.340+0000 7f833de28700 1 -- 172.21.15.94:0/3936046713 wait complete. 2023-09-30T00:17:45.344 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.341+0000 7f833de28700 1 Processor -- start 2023-09-30T00:17:45.344 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.341+0000 7f833de28700 1 -- start start 2023-09-30T00:17:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.341+0000 7f833de28700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8338070c50 0x7f8338144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.341+0000 7f833de28700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 0x7f83380ac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.341+0000 7f833de28700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f833813fd70 0x7f83380acee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.341+0000 7f833de28700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380ad420 con 0x7f8338070c50 2023-09-30T00:17:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.341+0000 7f833de28700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380aa800 con 0x7f833813fd70 2023-09-30T00:17:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.341+0000 7f833de28700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380aaac0 con 0x7f83380b7160 2023-09-30T00:17:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.341+0000 7f83377fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8338070c50 0x7f8338144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.347 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.341+0000 7f83377fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8338070c50 0x7f8338144190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33322/0 (socket says 172.21.15.94:33322) 2023-09-30T00:17:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.341+0000 7f83377fe700 1 -- 172.21.15.94:0/3091105011 learned_addr learned my addr 172.21.15.94:0/3091105011 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.341+0000 7f8336ffd700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 0x7f83380ac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.342+0000 7f83377fe700 1 -- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 msgr2=0x7f83380ac9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.342+0000 7f83377fe700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 0x7f83380ac9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.342+0000 7f83377fe700 1 -- 172.21.15.94:0/3091105011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f833813fd70 msgr2=0x7f83380acee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.342+0000 7f83377fe700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f833813fd70 0x7f83380acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.342+0000 7f83377fe700 1 -- 172.21.15.94:0/3091105011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c009420 con 0x7f8338070c50 2023-09-30T00:17:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.342+0000 7f8336ffd700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 0x7f83380ac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.342+0000 7f83377fe700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8338070c50 0x7f8338144190 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f83380a9730 tx=0x7f8328017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.342+0000 7f831f7fe700 1 -- 172.21.15.94:0/3091105011 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832800bbb0 con 0x7f8338070c50 2023-09-30T00:17:45.348 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.342+0000 7f831f7fe700 1 -- 172.21.15.94:0/3091105011 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8328011420 con 0x7f8338070c50 2023-09-30T00:17:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.342+0000 7f831f7fe700 1 -- 172.21.15.94:0/3091105011 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83280117a0 con 0x7f8338070c50 2023-09-30T00:17:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.342+0000 7f833de28700 1 -- 172.21.15.94:0/3091105011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8328009420 con 0x7f8338070c50 2023-09-30T00:17:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.342+0000 7f833de28700 1 -- 172.21.15.94:0/3091105011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8338144aa0 con 0x7f8338070c50 2023-09-30T00:17:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.343+0000 7f833de28700 1 -- 172.21.15.94:0/3091105011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83380a0dd0 con 0x7f8338070c50 2023-09-30T00:17:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.344+0000 7f831f7fe700 1 -- 172.21.15.94:0/3091105011 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) v1 ==== 126883+0+0 (secure 0 0 0) 0x7f8328011b50 con 0x7f8338070c50 2023-09-30T00:17:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.344+0000 7f831f7fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2023-09-30T00:17:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.344+0000 7f831f7fe700 4 mgrc handle_mgr_map Got map version 4 2023-09-30T00:17:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.344+0000 7f831f7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.344+0000 7f831f7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:45.349 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.344+0000 7f831f7fe700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8324098720 0x7f832409abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.344+0000 7f8336ffd700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8324098720 0x7f832409abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.344+0000 7f831f7fe700 1 -- 172.21.15.94:0/3091105011 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3092+0+0 (secure 0 0 0) 0x7f83280c1680 con 0x7f8338070c50 2023-09-30T00:17:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.345+0000 7f8336ffd700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8324098720 0x7f832409abd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f833813c760 tx=0x7f832c0091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:45.350 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.349+0000 7f831f7fe700 1 -- 172.21.15.94:0/3091105011 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f832808f370 con 0x7f8338070c50 2023-09-30T00:17:45.533 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.533+0000 7f833de28700 1 -- 172.21.15.94:0/3091105011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f833813e410 con 0x7f8338070c50 2023-09-30T00:17:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.534+0000 7f831f7fe700 1 -- 172.21.15.94:0/3091105011 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7447 (secure 0 0 0) 0x7f8328092990 con 0x7f8338070c50 2023-09-30T00:17:45.534 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:45.534 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":12,"fsid":"3c891173-587e-4c6f-9c6a-04ec750eedad","created":"2023-09-30T00:17:34.347592+0000","modified":"2023-09-30T00:17:45.187905+0000","last_up_change":"2023-09-30T00:17:42.879328+0000","last_in_change":"2023-09-30T00:17:35.583427+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T00:17:43.481613+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c508b402-f8ce-4461-823b-9e35fcd8d40f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":84364},{"type":"v1","addr":"172.21.15.94:6801","nonce":84364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":84364},{"type":"v1","addr":"172.21.15.94:6803","nonce":84364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":84364},{"type":"v1","addr":"172.21.15.94:6807","nonce":84364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":84364},{"type":"v1","addr":"172.21.15.94:6805","nonce":84364}]},"public_addr":"172.21.15.94:6801/84364","cluster_addr":"172.21.15.94:6803/84364","heartbeat_back_addr":"172.21.15.94:6807/84364","heartbeat_front_addr":"172.21.15.94:6805/84364","state":["exists","up"]},{"osd":1,"uuid":"e96ecdf0-bb38-4eb1-be99-54f813c12888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":86850},{"type":"v1","addr":"172.21.15.121:6817","nonce":86850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":86850},{"type":"v1","addr":"172.21.15.121:6819","nonce":86850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":86850},{"type":"v1","addr":"172.21.15.121:6823","nonce":86850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":86850},{"type":"v1","addr":"172.21.15.121:6821","nonce":86850}]},"public_addr":"172.21.15.121:6817/86850","cluster_addr":"172.21.15.121:6819/86850","heartbeat_back_addr":"172.21.15.121:6823/86850","heartbeat_front_addr":"172.21.15.121:6821/86850","state":["exists","up"]},{"osd":2,"uuid":"68700ae4-764c-4dff-9cf2-5f33132cd0d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":86820},{"type":"v1","addr":"172.21.15.121:6801","nonce":86820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":86820},{"type":"v1","addr":"172.21.15.121:6803","nonce":86820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":86820},{"type":"v1","addr":"172.21.15.121:6810","nonce":86820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":86820},{"type":"v1","addr":"172.21.15.121:6806","nonce":86820}]},"public_addr":"172.21.15.121:6801/86820","cluster_addr":"172.21.15.121:6803/86820","heartbeat_back_addr":"172.21.15.121:6810/86820","heartbeat_front_addr":"172.21.15.121:6806/86820","state":["exists","up"]},{"osd":3,"uuid":"54f17ec8-2561-42e8-bbd7-262130ffdac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":86821},{"type":"v1","addr":"172.21.15.121:6807","nonce":86821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":86821},{"type":"v1","addr":"172.21.15.121:6811","nonce":86821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":86821},{"type":"v1","addr":"172.21.15.121:6815","nonce":86821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":86821},{"type":"v1","addr":"172.21.15.121:6813","nonce":86821}]},"public_addr":"172.21.15.121:6807/86821","cluster_addr":"172.21.15.121:6811/86821","heartbeat_back_addr":"172.21.15.121:6815/86821","heartbeat_front_addr":"172.21.15.121:6813/86821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:39.951865+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.769149+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.738994+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.748197+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:17:45.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.535+0000 7f833de28700 10 mgrc shutdown 2023-09-30T00:17:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.535+0000 7f833de28700 1 -- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8324098720 msgr2=0x7f832409abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.535+0000 7f833de28700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8324098720 0x7f832409abd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f833813c760 tx=0x7f832c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.535+0000 7f833de28700 1 -- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8338070c50 msgr2=0x7f8338144190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.535+0000 7f833de28700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8338070c50 0x7f8338144190 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f83380a9730 tx=0x7f8328017000 comp rx=0 tx=0).stop 2023-09-30T00:17:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.536+0000 7f833de28700 1 -- 172.21.15.94:0/3091105011 shutdown_connections 2023-09-30T00:17:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.536+0000 7f833de28700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8324098720 0x7f832409abd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.536+0000 7f833de28700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f833813fd70 0x7f83380acee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.536+0000 7f833de28700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8338070c50 0x7f8338144190 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.536+0000 7f833de28700 1 --2- 172.21.15.94:0/3091105011 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83380b7160 0x7f83380ac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.536+0000 7f833de28700 1 -- 172.21.15.94:0/3091105011 >> 172.21.15.94:0/3091105011 conn(0x7f833806a090 msgr2=0x7f83380b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:45.538 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.536+0000 7f833de28700 1 -- 172.21.15.94:0/3091105011 shutdown_connections 2023-09-30T00:17:45.538 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.536+0000 7f833de28700 1 -- 172.21.15.94:0/3091105011 wait complete. 2023-09-30T00:17:45.548 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-09-30T00:17:43.481613+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-09-30T00:17:45.548 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-09-30T00:17:45.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.664+0000 7f7b75f42700 1 Processor -- start 2023-09-30T00:17:45.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.664+0000 7f7b75f42700 1 -- start start 2023-09-30T00:17:45.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.664+0000 7f7b75f42700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7b700b52b0 0x7f7b70141930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.664+0000 7f7b75f42700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b700a7b70 0x7f7b701422b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.664+0000 7f7b75f42700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b70141e70 con 0x7f7b700a51a0 2023-09-30T00:17:45.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.664+0000 7f7b75f42700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b700b5680 con 0x7f7b700b52b0 2023-09-30T00:17:45.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.664+0000 7f7b75f42700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b701427f0 con 0x7f7b700a7b70 2023-09-30T00:17:45.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b6ffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b700a7b70 0x7f7b701422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b75741700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7b700a51a0 0x7f7b701343f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:32844/0 (socket says 172.21.15.94:32844) 2023-09-30T00:17:45.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b75741700 1 -- 172.21.15.94:0/36334486 learned_addr learned my addr 172.21.15.94:0/36334486 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:45.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b74f40700 1 --2- 172.21.15.94:0/36334486 >> v2:172.21.15.121:3300/0 conn(0x7f7b700b52b0 0x7f7b70141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b6f7fe700 1 -- 172.21.15.94:0/36334486 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542793729 0 0) 0x7f7b60001110 con 0x7f7b700a51a0 2023-09-30T00:17:45.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b6f7fe700 1 -- 172.21.15.94:0/36334486 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b5c003570 con 0x7f7b700a51a0 2023-09-30T00:17:45.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b6ffff700 1 -- 172.21.15.94:0/36334486 >> v2:172.21.15.121:3300/0 conn(0x7f7b700b52b0 msgr2=0x7f7b70141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:45.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b6ffff700 1 --2- 172.21.15.94:0/36334486 >> v2:172.21.15.121:3300/0 conn(0x7f7b700b52b0 0x7f7b70141930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b6ffff700 1 -- 172.21.15.94:0/36334486 >> v1:172.21.15.94:6789/0 conn(0x7f7b700a51a0 legacy=0x7f7b701343f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b6ffff700 1 -- 172.21.15.94:0/36334486 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b700a5710 con 0x7f7b700a7b70 2023-09-30T00:17:45.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b74f40700 1 --2- 172.21.15.94:0/36334486 >> v2:172.21.15.121:3300/0 conn(0x7f7b700b52b0 0x7f7b70141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:17:45.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b6ffff700 1 --2- 172.21.15.94:0/36334486 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b700a7b70 0x7f7b701422b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7b580098e0 tx=0x7f7b580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c4acf9fd5e0391b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:45.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b6f7fe700 1 -- 172.21.15.94:0/36334486 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5800b640 con 0x7f7b700a7b70 2023-09-30T00:17:45.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b6f7fe700 1 -- 172.21.15.94:0/36334486 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5800b7a0 con 0x7f7b700a7b70 2023-09-30T00:17:45.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.665+0000 7f7b6f7fe700 1 -- 172.21.15.94:0/36334486 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5800bb40 con 0x7f7b700a7b70 2023-09-30T00:17:45.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.666+0000 7f7b75f42700 1 -- 172.21.15.94:0/36334486 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b700a7b70 msgr2=0x7f7b701422b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.666+0000 7f7b75f42700 1 --2- 172.21.15.94:0/36334486 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b700a7b70 0x7f7b701422b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7b580098e0 tx=0x7f7b580091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.666+0000 7f7b75f42700 1 -- 172.21.15.94:0/36334486 shutdown_connections 2023-09-30T00:17:45.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.666+0000 7f7b75f42700 1 --2- 172.21.15.94:0/36334486 >> v2:172.21.15.121:3300/0 conn(0x7f7b700b52b0 0x7f7b70141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.666+0000 7f7b75f42700 1 --2- 172.21.15.94:0/36334486 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b700a7b70 0x7f7b701422b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.666+0000 7f7b75f42700 1 -- 172.21.15.94:0/36334486 >> 172.21.15.94:0/36334486 conn(0x7f7b7006c940 msgr2=0x7f7b7006cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:45.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.666+0000 7f7b75f42700 1 -- 172.21.15.94:0/36334486 shutdown_connections 2023-09-30T00:17:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.666+0000 7f7b75f42700 1 -- 172.21.15.94:0/36334486 wait complete. 2023-09-30T00:17:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.667+0000 7f7b75f42700 1 Processor -- start 2023-09-30T00:17:45.670 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.667+0000 7f7b75f42700 1 -- start start 2023-09-30T00:17:45.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.667+0000 7f7b75f42700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b700a51a0 0x7f7b70132820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.667+0000 7f7b75f42700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b700a7b70 0x7f7b7014fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.667+0000 7f7b75f42700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b700b52b0 0x7f7b700acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.667+0000 7f7b75f42700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b70133200 con 0x7f7b700a51a0 2023-09-30T00:17:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.667+0000 7f7b75f42700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b701334e0 con 0x7f7b700a7b70 2023-09-30T00:17:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.667+0000 7f7b75f42700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b70150530 con 0x7f7b700b52b0 2023-09-30T00:17:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.667+0000 7f7b74f40700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b700a51a0 0x7f7b70132820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b74f40700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b700a51a0 0x7f7b70132820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33326/0 (socket says 172.21.15.94:33326) 2023-09-30T00:17:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b74f40700 1 -- 172.21.15.94:0/906982022 learned_addr learned my addr 172.21.15.94:0/906982022 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b75741700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b700b52b0 0x7f7b700acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b6ffff700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b700a7b70 0x7f7b7014fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b74f40700 1 -- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b700b52b0 msgr2=0x7f7b700acfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b74f40700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b700b52b0 0x7f7b700acfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b74f40700 1 -- 172.21.15.94:0/906982022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b700a7b70 msgr2=0x7f7b7014fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b74f40700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b700a7b70 0x7f7b7014fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b74f40700 1 -- 172.21.15.94:0/906982022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58009420 con 0x7f7b700a51a0 2023-09-30T00:17:45.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b74f40700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b700a51a0 0x7f7b70132820 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800cfb0 tx=0x7f7b68007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:45.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b6cff9700 1 -- 172.21.15.94:0/906982022 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6800ad80 con 0x7f7b700a51a0 2023-09-30T00:17:45.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b75f42700 1 -- 172.21.15.94:0/906982022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b701507f0 con 0x7f7b700a51a0 2023-09-30T00:17:45.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b75f42700 1 -- 172.21.15.94:0/906982022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7014d7d0 con 0x7f7b700a51a0 2023-09-30T00:17:45.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b6cff9700 1 -- 172.21.15.94:0/906982022 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b68016630 con 0x7f7b700a51a0 2023-09-30T00:17:45.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.668+0000 7f7b6cff9700 1 -- 172.21.15.94:0/906982022 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b680167a0 con 0x7f7b700a51a0 2023-09-30T00:17:45.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.669+0000 7f7b75f42700 1 -- 172.21.15.94:0/906982022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7004c380 con 0x7f7b700a51a0 2023-09-30T00:17:45.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.670+0000 7f7b6cff9700 1 -- 172.21.15.94:0/906982022 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) v1 ==== 126883+0+0 (secure 0 0 0) 0x7f7b68016bd0 con 0x7f7b700a51a0 2023-09-30T00:17:45.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.670+0000 7f7b6cff9700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2023-09-30T00:17:45.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.670+0000 7f7b6cff9700 4 mgrc handle_mgr_map Got map version 4 2023-09-30T00:17:45.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.670+0000 7f7b6cff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:45.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.670+0000 7f7b6cff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:45.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.670+0000 7f7b6cff9700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b50098670 0x7f7b5009ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.670+0000 7f7b6ffff700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b50098670 0x7f7b5009ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.671+0000 7f7b6cff9700 1 -- 172.21.15.94:0/906982022 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3092+0+0 (secure 0 0 0) 0x7f7b68012070 con 0x7f7b700a51a0 2023-09-30T00:17:45.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.671+0000 7f7b6ffff700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b50098670 0x7f7b5009ab20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7b580093f0 tx=0x7f7b58017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:45.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.675+0000 7f7b6cff9700 1 -- 172.21.15.94:0/906982022 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b6808e330 con 0x7f7b700a51a0 2023-09-30T00:17:45.857 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.857+0000 7f7b75f42700 1 -- 172.21.15.94:0/906982022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7b70132670 con 0x7f7b700a51a0 2023-09-30T00:17:45.858 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.857+0000 7f7b6cff9700 1 -- 172.21.15.94:0/906982022 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v12) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7b68091950 con 0x7f7b700a51a0 2023-09-30T00:17:45.858 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 8 2023-09-30T00:17:45.860 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 10 mgrc shutdown 2023-09-30T00:17:45.860 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 1 -- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b50098670 msgr2=0x7f7b5009ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.860 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b50098670 0x7f7b5009ab20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7b580093f0 tx=0x7f7b58017000 comp rx=0 tx=0).stop 2023-09-30T00:17:45.860 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 1 -- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b700a51a0 msgr2=0x7f7b70132820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.860 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b700a51a0 0x7f7b70132820 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800cfb0 tx=0x7f7b68007870 comp rx=0 tx=0).stop 2023-09-30T00:17:45.860 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 1 -- 172.21.15.94:0/906982022 shutdown_connections 2023-09-30T00:17:45.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b50098670 0x7f7b5009ab20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b700b52b0 0x7f7b700acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b700a51a0 0x7f7b70132820 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 1 --2- 172.21.15.94:0/906982022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b700a7b70 0x7f7b7014fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 1 -- 172.21.15.94:0/906982022 >> 172.21.15.94:0/906982022 conn(0x7f7b7006c940 msgr2=0x7f7b700b1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:45.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 1 -- 172.21.15.94:0/906982022 shutdown_connections 2023-09-30T00:17:45.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.859+0000 7f7b75f42700 1 -- 172.21.15.94:0/906982022 wait complete. 2023-09-30T00:17:45.872 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-30T00:17:45.985 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.984+0000 7ff294359700 1 Processor -- start 2023-09-30T00:17:45.985 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.984+0000 7ff294359700 1 -- start start 2023-09-30T00:17:45.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.985+0000 7ff294359700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff28c06a460 0x7ff28c0b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.985+0000 7ff294359700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0b5680 0x7ff28c14b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.985+0000 7ff294359700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28c069760 con 0x7ff28c0a46c0 2023-09-30T00:17:45.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.985+0000 7ff294359700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff28c0a57f0 con 0x7ff28c06a460 2023-09-30T00:17:45.988 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.985+0000 7ff294359700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c0a4a90 con 0x7ff28c0b5680 2023-09-30T00:17:45.988 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.985+0000 7ff2928f6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0b5680 0x7ff28c14b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.988 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.985+0000 7ff2920f5700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff28c0a46c0 0x7ff28c068c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:32848/0 (socket says 172.21.15.94:32848) 2023-09-30T00:17:45.988 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.985+0000 7ff2920f5700 1 -- 172.21.15.94:0/1016081045 learned_addr learned my addr 172.21.15.94:0/1016081045 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:45.988 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.985+0000 7ff2918f4700 1 --2- 172.21.15.94:0/1016081045 >> v2:172.21.15.94:3300/0 conn(0x7ff28c06a460 0x7ff28c0b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.988 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.986+0000 7ff2928f6700 1 -- 172.21.15.94:0/1016081045 >> v2:172.21.15.94:3300/0 conn(0x7ff28c06a460 msgr2=0x7ff28c0b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:45.988 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.986+0000 7ff2928f6700 1 --2- 172.21.15.94:0/1016081045 >> v2:172.21.15.94:3300/0 conn(0x7ff28c06a460 0x7ff28c0b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.988 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.986+0000 7ff2910f3700 1 -- 172.21.15.94:0/1016081045 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419858687 0 0) 0x7ff28c069760 con 0x7ff28c0a46c0 2023-09-30T00:17:45.988 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.986+0000 7ff2928f6700 1 -- 172.21.15.94:0/1016081045 >> v1:172.21.15.94:6789/0 conn(0x7ff28c0a46c0 legacy=0x7ff28c068c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.988 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.986+0000 7ff2928f6700 1 -- 172.21.15.94:0/1016081045 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c069c80 con 0x7ff28c0b5680 2023-09-30T00:17:45.988 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.986+0000 7ff2928f6700 1 --2- 172.21.15.94:0/1016081045 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0b5680 0x7ff28c14b060 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff284009910 tx=0x7ff2840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97c3f5d32572b0db server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:45.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.986+0000 7ff2910f3700 1 -- 172.21.15.94:0/1016081045 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28400b640 con 0x7ff28c0b5680 2023-09-30T00:17:45.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.986+0000 7ff2910f3700 1 -- 172.21.15.94:0/1016081045 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28400b7a0 con 0x7ff28c0b5680 2023-09-30T00:17:45.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.986+0000 7ff2910f3700 1 -- 172.21.15.94:0/1016081045 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28400bb60 con 0x7ff28c0b5680 2023-09-30T00:17:45.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.986+0000 7ff294359700 1 -- 172.21.15.94:0/1016081045 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0b5680 msgr2=0x7ff28c14b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.986+0000 7ff294359700 1 --2- 172.21.15.94:0/1016081045 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0b5680 0x7ff28c14b060 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff284009910 tx=0x7ff2840091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.987+0000 7ff294359700 1 -- 172.21.15.94:0/1016081045 shutdown_connections 2023-09-30T00:17:45.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.987+0000 7ff294359700 1 --2- 172.21.15.94:0/1016081045 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0b5680 0x7ff28c14b060 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.987+0000 7ff294359700 1 --2- 172.21.15.94:0/1016081045 >> v2:172.21.15.94:3300/0 conn(0x7ff28c06a460 0x7ff28c0b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.987+0000 7ff294359700 1 -- 172.21.15.94:0/1016081045 >> 172.21.15.94:0/1016081045 conn(0x7ff28c06cbf0 msgr2=0x7ff28c06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:45.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.987+0000 7ff294359700 1 -- 172.21.15.94:0/1016081045 shutdown_connections 2023-09-30T00:17:45.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.987+0000 7ff294359700 1 -- 172.21.15.94:0/1016081045 wait complete. 2023-09-30T00:17:45.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.987+0000 7ff294359700 1 Processor -- start 2023-09-30T00:17:45.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff294359700 1 -- start start 2023-09-30T00:17:45.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff294359700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff28c06a460 0x7ff28c11d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff294359700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0a46c0 0x7ff28c140c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff294359700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff28c0b5680 0x7ff28c141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff294359700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c1416d0 con 0x7ff28c06a460 2023-09-30T00:17:45.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff294359700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c1418e0 con 0x7ff28c0b5680 2023-09-30T00:17:45.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff294359700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c0adb70 con 0x7ff28c0a46c0 2023-09-30T00:17:45.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff2918f4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0a46c0 0x7ff28c140c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff2920f5700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff28c06a460 0x7ff28c11d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff2918f4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0a46c0 0x7ff28c140c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51450/0 (socket says 172.21.15.94:51450) 2023-09-30T00:17:45.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff2918f4700 1 -- 172.21.15.94:0/3924392870 learned_addr learned my addr 172.21.15.94:0/3924392870 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:45.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff2920f5700 1 -- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0a46c0 msgr2=0x7ff28c140c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff2920f5700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0a46c0 0x7ff28c140c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff2920f5700 1 -- 172.21.15.94:0/3924392870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff28c0b5680 msgr2=0x7ff28c141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:45.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff2920f5700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff28c0b5680 0x7ff28c141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:45.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff2920f5700 1 -- 172.21.15.94:0/3924392870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff284009420 con 0x7ff28c06a460 2023-09-30T00:17:45.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.988+0000 7ff2918f4700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0a46c0 0x7ff28c140c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:45.994 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.989+0000 7ff2920f5700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff28c06a460 0x7ff28c11d6b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff28800cc90 tx=0x7ff288001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:45.994 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.989+0000 7ff2827fc700 1 -- 172.21.15.94:0/3924392870 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28800a920 con 0x7ff28c06a460 2023-09-30T00:17:45.994 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.989+0000 7ff2827fc700 1 -- 172.21.15.94:0/3924392870 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28800aa80 con 0x7ff28c06a460 2023-09-30T00:17:45.994 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.989+0000 7ff2827fc700 1 -- 172.21.15.94:0/3924392870 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28800ae20 con 0x7ff28c06a460 2023-09-30T00:17:45.994 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.989+0000 7ff294359700 1 -- 172.21.15.94:0/3924392870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28c1400c0 con 0x7ff28c06a460 2023-09-30T00:17:45.994 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.989+0000 7ff294359700 1 -- 172.21.15.94:0/3924392870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28c0af2e0 con 0x7ff28c06a460 2023-09-30T00:17:45.994 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.990+0000 7ff294359700 1 -- 172.21.15.94:0/3924392870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28c06fbd0 con 0x7ff28c06a460 2023-09-30T00:17:45.994 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.991+0000 7ff2827fc700 1 -- 172.21.15.94:0/3924392870 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) v1 ==== 126883+0+0 (secure 0 0 0) 0x7ff288007300 con 0x7ff28c06a460 2023-09-30T00:17:45.994 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.991+0000 7ff2827fc700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2023-09-30T00:17:45.994 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.991+0000 7ff2827fc700 4 mgrc handle_mgr_map Got map version 4 2023-09-30T00:17:45.994 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.991+0000 7ff2827fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:45.994 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.991+0000 7ff2827fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:45.995 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.991+0000 7ff2827fc700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff2700986f0 0x7ff27009aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:45.995 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.991+0000 7ff2918f4700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff2700986f0 0x7ff27009aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:45.995 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.991+0000 7ff2827fc700 1 -- 172.21.15.94:0/3924392870 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3092+0+0 (secure 0 0 0) 0x7ff2880c0d10 con 0x7ff28c06a460 2023-09-30T00:17:45.995 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.992+0000 7ff2918f4700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff2700986f0 0x7ff27009aba0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0af210 tx=0x7ff27c0091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:45.995 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:45.995+0000 7ff2827fc700 1 -- 172.21.15.94:0/3924392870 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff28808ea00 con 0x7ff28c06a460 2023-09-30T00:17:46.219 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.219+0000 7ff294359700 1 -- 172.21.15.94:0/3924392870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff28c06ecb0 con 0x7ff28c06a460 2023-09-30T00:17:46.220 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.219+0000 7ff2827fc700 1 -- 172.21.15.94:0/3924392870 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1256 (secure 0 0 0) 0x7ff288092020 con 0x7ff28c06a460 2023-09-30T00:17:46.220 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:46.220 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6811/2461396790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2461396790},{"type":"v1","addr":"172.21.15.94:6811","nonce":2461396790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-09-30T00:17:46.222 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 10 mgrc shutdown 2023-09-30T00:17:46.222 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 1 -- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff2700986f0 msgr2=0x7ff27009aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:46.222 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff2700986f0 0x7ff27009aba0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0af210 tx=0x7ff27c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:46.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 1 -- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff28c06a460 msgr2=0x7ff28c11d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:46.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff28c06a460 0x7ff28c11d6b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff28800cc90 tx=0x7ff288001a20 comp rx=0 tx=0).stop 2023-09-30T00:17:46.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 1 -- 172.21.15.94:0/3924392870 shutdown_connections 2023-09-30T00:17:46.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff2700986f0 0x7ff27009aba0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:46.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff28c0b5680 0x7ff28c141190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:46.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff28c06a460 0x7ff28c11d6b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:46.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 1 --2- 172.21.15.94:0/3924392870 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff28c0a46c0 0x7ff28c140c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:46.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 1 -- 172.21.15.94:0/3924392870 >> 172.21.15.94:0/3924392870 conn(0x7ff28c06cbf0 msgr2=0x7ff28c06cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:46.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 1 -- 172.21.15.94:0/3924392870 shutdown_connections 2023-09-30T00:17:46.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:46.221+0000 7ff294359700 1 -- 172.21.15.94:0/3924392870 wait complete. 2023-09-30T00:17:46.223 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 2 2023-09-30T00:17:48.234 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-30T00:17:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.349+0000 7fa875253700 1 Processor -- start 2023-09-30T00:17:48.350 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.349+0000 7fa875253700 1 -- start start 2023-09-30T00:17:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa875253700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8700b7160 0x7fa8700a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa875253700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa87013fd70 0x7fa8701401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa875253700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8700a1310 con 0x7fa870070c50 2023-09-30T00:17:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa875253700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa870071020 con 0x7fa8700b7160 2023-09-30T00:17:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa875253700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8700b76a0 con 0x7fa87013fd70 2023-09-30T00:17:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86659c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8700b7160 0x7fa8700a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:48.355 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86f59e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa87013fd70 0x7fa8701401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86ed9d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa870070c50 0x7fa8700a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:32862/0 (socket says 172.21.15.94:32862) 2023-09-30T00:17:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86f59e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa87013fd70 0x7fa8701401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51490/0 (socket says 172.21.15.94:51490) 2023-09-30T00:17:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86659c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8700b7160 0x7fa8700a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33404/0 (socket says 172.21.15.94:33404) 2023-09-30T00:17:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86ed9d700 1 -- 172.21.15.94:0/1216773378 learned_addr learned my addr 172.21.15.94:0/1216773378 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86659c700 1 -- 172.21.15.94:0/1216773378 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa87013fd70 msgr2=0x7fa8701401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86659c700 1 --2- 172.21.15.94:0/1216773378 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa87013fd70 0x7fa8701401d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86659c700 1 -- 172.21.15.94:0/1216773378 >> v1:172.21.15.94:6789/0 conn(0x7fa870070c50 legacy=0x7fa8700a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86e59c700 1 -- 172.21.15.94:0/1216773378 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093696618 0 0) 0x7fa860001110 con 0x7fa870070c50 2023-09-30T00:17:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86659c700 1 -- 172.21.15.94:0/1216773378 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8700b7940 con 0x7fa8700b7160 2023-09-30T00:17:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86f59e700 1 --2- 172.21.15.94:0/1216773378 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa87013fd70 0x7fa8701401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86659c700 1 --2- 172.21.15.94:0/1216773378 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8700b7160 0x7fa8700a0dd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8580098e0 tx=0x7fa8580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f08447d13711acc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86e59c700 1 -- 172.21.15.94:0/1216773378 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85800b6d0 con 0x7fa8700b7160 2023-09-30T00:17:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86e59c700 1 -- 172.21.15.94:0/1216773378 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85800b830 con 0x7fa8700b7160 2023-09-30T00:17:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.350+0000 7fa86e59c700 1 -- 172.21.15.94:0/1216773378 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85800bbd0 con 0x7fa8700b7160 2023-09-30T00:17:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.351+0000 7fa875253700 1 -- 172.21.15.94:0/1216773378 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8700b7160 msgr2=0x7fa8700a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.351+0000 7fa875253700 1 --2- 172.21.15.94:0/1216773378 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8700b7160 0x7fa8700a0dd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8580098e0 tx=0x7fa8580091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.351+0000 7fa875253700 1 -- 172.21.15.94:0/1216773378 shutdown_connections 2023-09-30T00:17:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.351+0000 7fa875253700 1 --2- 172.21.15.94:0/1216773378 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa87013fd70 0x7fa8701401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.351+0000 7fa875253700 1 --2- 172.21.15.94:0/1216773378 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8700b7160 0x7fa8700a0dd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.351+0000 7fa875253700 1 -- 172.21.15.94:0/1216773378 >> 172.21.15.94:0/1216773378 conn(0x7fa87006a090 msgr2=0x7fa8700702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.351+0000 7fa875253700 1 -- 172.21.15.94:0/1216773378 shutdown_connections 2023-09-30T00:17:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.351+0000 7fa875253700 1 -- 172.21.15.94:0/1216773378 wait complete. 2023-09-30T00:17:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.352+0000 7fa875253700 1 Processor -- start 2023-09-30T00:17:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.352+0000 7fa875253700 1 -- start start 2023-09-30T00:17:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa875253700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa870070c50 0x7fa8700a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa875253700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8700b7160 0x7fa8700a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa875253700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa87013fd70 0x7fa87009b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa875253700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8700a4290 con 0x7fa87013fd70 2023-09-30T00:17:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa875253700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8700a4530 con 0x7fa870070c50 2023-09-30T00:17:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa875253700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa870136090 con 0x7fa8700b7160 2023-09-30T00:17:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86659c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8700b7160 0x7fa8700a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86f59e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa87013fd70 0x7fa87009b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86659c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8700b7160 0x7fa8700a6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51496/0 (socket says 172.21.15.94:51496) 2023-09-30T00:17:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86f59e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa87013fd70 0x7fa87009b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33406/0 (socket says 172.21.15.94:33406) 2023-09-30T00:17:48.359 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86659c700 1 -- 172.21.15.94:0/3480997739 learned_addr learned my addr 172.21.15.94:0/3480997739 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:48.360 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86ed9d700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa870070c50 0x7fa8700a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:48.360 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86659c700 1 -- 172.21.15.94:0/3480997739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa870070c50 msgr2=0x7fa8700a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:48.360 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86659c700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa870070c50 0x7fa8700a3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.360 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86659c700 1 -- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa87013fd70 msgr2=0x7fa87009b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:48.360 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86659c700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa87013fd70 0x7fa87009b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.360 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86659c700 1 -- 172.21.15.94:0/3480997739 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa868009a20 con 0x7fa8700b7160 2023-09-30T00:17:48.360 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86ed9d700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa870070c50 0x7fa8700a3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:17:48.360 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa86659c700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8700b7160 0x7fa8700a6850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa8700a31d0 tx=0x7fa858017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:48.360 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa8677fe700 1 -- 172.21.15.94:0/3480997739 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85800bbb0 con 0x7fa8700b7160 2023-09-30T00:17:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa8677fe700 1 -- 172.21.15.94:0/3480997739 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa858011420 con 0x7fa8700b7160 2023-09-30T00:17:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa875253700 1 -- 172.21.15.94:0/3480997739 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa858009420 con 0x7fa8700b7160 2023-09-30T00:17:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa8677fe700 1 -- 172.21.15.94:0/3480997739 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8580117a0 con 0x7fa8700b7160 2023-09-30T00:17:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.353+0000 7fa875253700 1 -- 172.21.15.94:0/3480997739 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa870143d70 con 0x7fa8700b7160 2023-09-30T00:17:48.361 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.354+0000 7fa875253700 1 -- 172.21.15.94:0/3480997739 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa870071630 con 0x7fa8700b7160 2023-09-30T00:17:48.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.355+0000 7fa8677fe700 1 -- 172.21.15.94:0/3480997739 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 4) v1 ==== 126883+0+0 (secure 0 0 0) 0x7fa858011b10 con 0x7fa8700b7160 2023-09-30T00:17:48.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.355+0000 7fa8677fe700 20 mgrc handle_mgr_map mgrmap(e 4) v1 2023-09-30T00:17:48.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.356+0000 7fa8677fe700 4 mgrc handle_mgr_map Got map version 4 2023-09-30T00:17:48.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.356+0000 7fa8677fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:48.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.356+0000 7fa8677fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:48.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.356+0000 7fa8677fe700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa8500986f0 0x7fa85009aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:48.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.356+0000 7fa86ed9d700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa8500986f0 0x7fa85009aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:48.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.356+0000 7fa8677fe700 1 -- 172.21.15.94:0/3480997739 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7fa8580c18d0 con 0x7fa8700b7160 2023-09-30T00:17:48.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.356+0000 7fa86ed9d700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa8500986f0 0x7fa85009aba0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa8600018c0 tx=0x7fa860009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:48.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.360+0000 7fa8677fe700 1 -- 172.21.15.94:0/3480997739 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa85808f440 con 0x7fa8700b7160 2023-09-30T00:17:48.566 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.566+0000 7fa8677fe700 1 -- 172.21.15.94:0/3480997739 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fa85808f810 con 0x7fa8700b7160 2023-09-30T00:17:48.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.566+0000 7fa8677fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:48.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.566+0000 7fa8677fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:48.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.566+0000 7fa8677fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:48.591 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.591+0000 7fa875253700 1 -- 172.21.15.94:0/3480997739 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa870144a60 con 0x7fa8700b7160 2023-09-30T00:17:48.591 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.591+0000 7fa8677fe700 1 -- 172.21.15.94:0/3480997739 <== mon.2 v2:172.21.15.94:3301/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2768 (secure 0 0 0) 0x7fa858092ed0 con 0x7fa8700b7160 2023-09-30T00:17:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:48.592 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4187,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6813/1048306399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":1048306399},{"type":"v1","addr":"172.21.15.94:6813","nonce":1048306399}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4231,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6825/549674001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":549674001},{"type":"v1","addr":"172.21.15.121:6825","nonce":549674001}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6811/2461396790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2461396790},{"type":"v1","addr":"172.21.15.94:6811","nonce":2461396790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-09-30T00:17:48.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 10 mgrc shutdown 2023-09-30T00:17:48.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 1 -- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa8500986f0 msgr2=0x7fa85009aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:48.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa8500986f0 0x7fa85009aba0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa8600018c0 tx=0x7fa860009040 comp rx=0 tx=0).stop 2023-09-30T00:17:48.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 1 -- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8700b7160 msgr2=0x7fa8700a6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:48.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8700b7160 0x7fa8700a6850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa8700a31d0 tx=0x7fa858017000 comp rx=0 tx=0).stop 2023-09-30T00:17:48.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 1 -- 172.21.15.94:0/3480997739 shutdown_connections 2023-09-30T00:17:48.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa8500986f0 0x7fa85009aba0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa87013fd70 0x7fa87009b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.595 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa870070c50 0x7fa8700a3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.595 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 1 --2- 172.21.15.94:0/3480997739 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8700b7160 0x7fa8700a6850 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.595 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 1 -- 172.21.15.94:0/3480997739 >> 172.21.15.94:0/3480997739 conn(0x7fa87006a090 msgr2=0x7fa8701393d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:48.595 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 1 -- 172.21.15.94:0/3480997739 shutdown_connections 2023-09-30T00:17:48.595 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.593+0000 7fa875253700 1 -- 172.21.15.94:0/3480997739 wait complete. 2023-09-30T00:17:48.595 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 3 2023-09-30T00:17:48.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:17:48.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.718+0000 7f5fcfc59700 1 Processor -- start 2023-09-30T00:17:48.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.718+0000 7f5fcfc59700 1 -- start start 2023-09-30T00:17:48.720 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.718+0000 7f5fcfc59700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 0x7f5fc80a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:48.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.718+0000 7f5fcfc59700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5fc813fd70 0x7f5fc81401f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:48.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.718+0000 7f5fcfc59700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc80a1310 con 0x7f5fc80b7160 2023-09-30T00:17:48.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.718+0000 7f5fcfc59700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc8071020 con 0x7f5fc813fd70 2023-09-30T00:17:48.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.718+0000 7f5fcfc59700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc80b76a0 con 0x7f5fc8070c50 2023-09-30T00:17:48.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fcd9f5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 0x7f5fc80a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:48.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fcd1f4700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5fc80b7160 0x7f5fc80a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:32868/0 (socket says 172.21.15.94:32868) 2023-09-30T00:17:48.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fcd1f4700 1 -- 172.21.15.94:0/4069907195 learned_addr learned my addr 172.21.15.94:0/4069907195 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:48.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fce1f6700 1 --2- 172.21.15.94:0/4069907195 >> v2:172.21.15.121:3300/0 conn(0x7f5fc813fd70 0x7f5fc81401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:48.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fcc9f3700 1 -- 172.21.15.94:0/4069907195 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368646873 0 0) 0x7f5fb8001110 con 0x7f5fc80b7160 2023-09-30T00:17:48.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fcd9f5700 1 -- 172.21.15.94:0/4069907195 >> v2:172.21.15.121:3300/0 conn(0x7f5fc813fd70 msgr2=0x7f5fc81401f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:48.722 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fcd9f5700 1 --2- 172.21.15.94:0/4069907195 >> v2:172.21.15.121:3300/0 conn(0x7f5fc813fd70 0x7f5fc81401f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.722 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fcd9f5700 1 -- 172.21.15.94:0/4069907195 >> v1:172.21.15.94:6789/0 conn(0x7f5fc80b7160 legacy=0x7f5fc80a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:48.722 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fcd9f5700 1 -- 172.21.15.94:0/4069907195 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc80b7940 con 0x7f5fc8070c50 2023-09-30T00:17:48.722 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fcd9f5700 1 --2- 172.21.15.94:0/4069907195 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 0x7f5fc80a0dd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40098e0 tx=0x7f5fc40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a1339ed40345816 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:48.722 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fcc9f3700 1 -- 172.21.15.94:0/4069907195 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc400b6d0 con 0x7f5fc8070c50 2023-09-30T00:17:48.722 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fcc9f3700 1 -- 172.21.15.94:0/4069907195 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc400b830 con 0x7f5fc8070c50 2023-09-30T00:17:48.722 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.719+0000 7f5fcc9f3700 1 -- 172.21.15.94:0/4069907195 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc400bbd0 con 0x7f5fc8070c50 2023-09-30T00:17:48.722 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.720+0000 7f5fcfc59700 1 -- 172.21.15.94:0/4069907195 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 msgr2=0x7f5fc80a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:48.722 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.720+0000 7f5fcfc59700 1 --2- 172.21.15.94:0/4069907195 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 0x7f5fc80a0dd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40098e0 tx=0x7f5fc40091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.722 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.720+0000 7f5fcfc59700 1 -- 172.21.15.94:0/4069907195 shutdown_connections 2023-09-30T00:17:48.723 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.720+0000 7f5fcfc59700 1 --2- 172.21.15.94:0/4069907195 >> v2:172.21.15.121:3300/0 conn(0x7f5fc813fd70 0x7f5fc81401f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.723 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.720+0000 7f5fcfc59700 1 --2- 172.21.15.94:0/4069907195 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 0x7f5fc80a0dd0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.723 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.720+0000 7f5fcfc59700 1 -- 172.21.15.94:0/4069907195 >> 172.21.15.94:0/4069907195 conn(0x7f5fc806a090 msgr2=0x7f5fc80702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:48.723 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.720+0000 7f5fcfc59700 1 -- 172.21.15.94:0/4069907195 shutdown_connections 2023-09-30T00:17:48.723 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.720+0000 7f5fcfc59700 1 -- 172.21.15.94:0/4069907195 wait complete. 2023-09-30T00:17:48.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fcfc59700 1 Processor -- start 2023-09-30T00:17:48.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fcfc59700 1 -- start start 2023-09-30T00:17:48.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fcfc59700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 0x7f5fc8143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:48.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fcfc59700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fc80b7160 0x7f5fc80a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:48.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fcfc59700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fc813fd70 0x7f5fc809c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:48.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fcfc59700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc80a5440 con 0x7f5fc813fd70 2023-09-30T00:17:48.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fcfc59700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc80a5700 con 0x7f5fc80b7160 2023-09-30T00:17:48.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fcfc59700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc8144be0 con 0x7f5fc8070c50 2023-09-30T00:17:48.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fcd9f5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 0x7f5fc8143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fce1f6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fc813fd70 0x7f5fc809c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fcd9f5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 0x7f5fc8143d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51520/0 (socket says 172.21.15.94:51520) 2023-09-30T00:17:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fce1f6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fc813fd70 0x7f5fc809c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33418/0 (socket says 172.21.15.94:33418) 2023-09-30T00:17:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.721+0000 7f5fcd9f5700 1 -- 172.21.15.94:0/1780633931 learned_addr learned my addr 172.21.15.94:0/1780633931 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fcd1f4700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fc80b7160 0x7f5fc80a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fcd9f5700 1 -- 172.21.15.94:0/1780633931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fc80b7160 msgr2=0x7f5fc80a44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fcd9f5700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fc80b7160 0x7f5fc80a44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fcd9f5700 1 -- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fc813fd70 msgr2=0x7f5fc809c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fcd9f5700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fc813fd70 0x7f5fc809c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fcd9f5700 1 -- 172.21.15.94:0/1780633931 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc4009420 con 0x7f5fc8070c50 2023-09-30T00:17:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fce1f6700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fc813fd70 0x7f5fc809c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fcd1f4700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fc80b7160 0x7f5fc80a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:17:48.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fcd9f5700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 0x7f5fc8143d80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5fc80a9730 tx=0x7f5fc4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:48.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fbdffb700 1 -- 172.21.15.94:0/1780633931 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc400bbb0 con 0x7f5fc8070c50 2023-09-30T00:17:48.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fbdffb700 1 -- 172.21.15.94:0/1780633931 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc4011420 con 0x7f5fc8070c50 2023-09-30T00:17:48.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fcfc59700 1 -- 172.21.15.94:0/1780633931 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc81442c0 con 0x7f5fc8070c50 2023-09-30T00:17:48.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fbdffb700 1 -- 172.21.15.94:0/1780633931 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc40117a0 con 0x7f5fc8070c50 2023-09-30T00:17:48.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.722+0000 7f5fcfc59700 1 -- 172.21.15.94:0/1780633931 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc813c550 con 0x7f5fc8070c50 2023-09-30T00:17:48.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.723+0000 7f5fcfc59700 1 -- 172.21.15.94:0/1780633931 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc813e4a0 con 0x7f5fc8070c50 2023-09-30T00:17:48.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.724+0000 7f5fbdffb700 1 -- 172.21.15.94:0/1780633931 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5fc4011b50 con 0x7f5fc8070c50 2023-09-30T00:17:48.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.724+0000 7f5fbdffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:48.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.725+0000 7f5fbdffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:48.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.725+0000 7f5fbdffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:48.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.725+0000 7f5fbdffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:48.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.725+0000 7f5fbdffb700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5fac098780 0x7f5fac09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:48.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.725+0000 7f5fcd1f4700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5fac098780 0x7f5fac09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:48.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.725+0000 7f5fbdffb700 1 -- 172.21.15.94:0/1780633931 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f5fc40c1960 con 0x7f5fc8070c50 2023-09-30T00:17:48.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.725+0000 7f5fcd1f4700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5fac098780 0x7f5fac09ac30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8143d30 tx=0x7f5fb8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:48.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.729+0000 7f5fbdffb700 1 -- 172.21.15.94:0/1780633931 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5fc408f3f0 con 0x7f5fc8070c50 2023-09-30T00:17:48.912 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.912+0000 7f5fcfc59700 1 -- 172.21.15.94:0/1780633931 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fc813e1c0 con 0x7f5fc8070c50 2023-09-30T00:17:48.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.913+0000 7f5fbdffb700 1 -- 172.21.15.94:0/1780633931 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+8981 (secure 0 0 0) 0x7f5fc402a090 con 0x7f5fc8070c50 2023-09-30T00:17:48.913 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:48.913 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":15,"fsid":"3c891173-587e-4c6f-9c6a-04ec750eedad","created":"2023-09-30T00:17:34.347592+0000","modified":"2023-09-30T00:17:48.198869+0000","last_up_change":"2023-09-30T00:17:42.879328+0000","last_in_change":"2023-09-30T00:17:35.583427+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T00:17:43.481613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T00:17:45.892643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c508b402-f8ce-4461-823b-9e35fcd8d40f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":84364},{"type":"v1","addr":"172.21.15.94:6801","nonce":84364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":84364},{"type":"v1","addr":"172.21.15.94:6803","nonce":84364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":84364},{"type":"v1","addr":"172.21.15.94:6807","nonce":84364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":84364},{"type":"v1","addr":"172.21.15.94:6805","nonce":84364}]},"public_addr":"172.21.15.94:6801/84364","cluster_addr":"172.21.15.94:6803/84364","heartbeat_back_addr":"172.21.15.94:6807/84364","heartbeat_front_addr":"172.21.15.94:6805/84364","state":["exists","up"]},{"osd":1,"uuid":"e96ecdf0-bb38-4eb1-be99-54f813c12888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":86850},{"type":"v1","addr":"172.21.15.121:6817","nonce":86850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":86850},{"type":"v1","addr":"172.21.15.121:6819","nonce":86850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":86850},{"type":"v1","addr":"172.21.15.121:6823","nonce":86850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":86850},{"type":"v1","addr":"172.21.15.121:6821","nonce":86850}]},"public_addr":"172.21.15.121:6817/86850","cluster_addr":"172.21.15.121:6819/86850","heartbeat_back_addr":"172.21.15.121:6823/86850","heartbeat_front_addr":"172.21.15.121:6821/86850","state":["exists","up"]},{"osd":2,"uuid":"68700ae4-764c-4dff-9cf2-5f33132cd0d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":86820},{"type":"v1","addr":"172.21.15.121:6801","nonce":86820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":86820},{"type":"v1","addr":"172.21.15.121:6803","nonce":86820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":86820},{"type":"v1","addr":"172.21.15.121:6810","nonce":86820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":86820},{"type":"v1","addr":"172.21.15.121:6806","nonce":86820}]},"public_addr":"172.21.15.121:6801/86820","cluster_addr":"172.21.15.121:6803/86820","heartbeat_back_addr":"172.21.15.121:6810/86820","heartbeat_front_addr":"172.21.15.121:6806/86820","state":["exists","up"]},{"osd":3,"uuid":"54f17ec8-2561-42e8-bbd7-262130ffdac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":86821},{"type":"v1","addr":"172.21.15.121:6807","nonce":86821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":86821},{"type":"v1","addr":"172.21.15.121:6811","nonce":86821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":86821},{"type":"v1","addr":"172.21.15.121:6815","nonce":86821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":86821},{"type":"v1","addr":"172.21.15.121:6813","nonce":86821}]},"public_addr":"172.21.15.121:6807/86821","cluster_addr":"172.21.15.121:6811/86821","heartbeat_back_addr":"172.21.15.121:6815/86821","heartbeat_front_addr":"172.21.15.121:6813/86821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:39.951865+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.769149+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.738994+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.748197+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:17:48.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.914+0000 7f5fcfc59700 10 mgrc shutdown 2023-09-30T00:17:48.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.914+0000 7f5fcfc59700 1 -- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5fac098780 msgr2=0x7f5fac09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:48.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.914+0000 7f5fcfc59700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5fac098780 0x7f5fac09ac30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8143d30 tx=0x7f5fb8009040 comp rx=0 tx=0).stop 2023-09-30T00:17:48.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.914+0000 7f5fcfc59700 1 -- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 msgr2=0x7f5fc8143d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:48.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.914+0000 7f5fcfc59700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 0x7f5fc8143d80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5fc80a9730 tx=0x7f5fc4017000 comp rx=0 tx=0).stop 2023-09-30T00:17:48.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.914+0000 7f5fcfc59700 1 -- 172.21.15.94:0/1780633931 shutdown_connections 2023-09-30T00:17:48.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.914+0000 7f5fcfc59700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5fac098780 0x7f5fac09ac30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.914+0000 7f5fcfc59700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5fc8070c50 0x7f5fc8143d80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.914+0000 7f5fcfc59700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fc813fd70 0x7f5fc809c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.914+0000 7f5fcfc59700 1 --2- 172.21.15.94:0/1780633931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fc80b7160 0x7f5fc80a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:48.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.915+0000 7f5fcfc59700 1 -- 172.21.15.94:0/1780633931 >> 172.21.15.94:0/1780633931 conn(0x7f5fc806a090 msgr2=0x7f5fc80a8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:48.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.915+0000 7f5fcfc59700 1 -- 172.21.15.94:0/1780633931 shutdown_connections 2023-09-30T00:17:48.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:48.915+0000 7f5fcfc59700 1 -- 172.21.15.94:0/1780633931 wait complete. 2023-09-30T00:17:48.927 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-09-30T00:17:43.481613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-09-30T00:17:45.892643+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-09-30T00:17:48.927 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-09-30T00:17:49.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.042+0000 7fe281716700 1 Processor -- start 2023-09-30T00:17:49.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.042+0000 7fe281716700 1 -- start start 2023-09-30T00:17:49.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.043+0000 7fe281716700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 0x7fe27c0a5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.043+0000 7fe281716700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe27c069d60 con 0x7fe27c069990 2023-09-30T00:17:49.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.043+0000 7fe281716700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe27c0a63e0 con 0x7fe27c0a49a0 2023-09-30T00:17:49.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.043+0000 7fe281716700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c06ca90 con 0x7fe27c0a5a50 2023-09-30T00:17:49.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.043+0000 7fe27affd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe27c069990 0x7fe27c06e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:32870/0 (socket says 172.21.15.94:32870) 2023-09-30T00:17:49.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.043+0000 7fe27affd700 1 -- 172.21.15.94:0/2040381055 learned_addr learned my addr 172.21.15.94:0/2040381055 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:49.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.043+0000 7fe27b7fe700 1 --2- 172.21.15.94:0/2040381055 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 0x7fe27c0a5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.043+0000 7fe27a7fc700 1 -- 172.21.15.94:0/2040381055 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498416317 0 0) 0x7fe27c069d60 con 0x7fe27c069990 2023-09-30T00:17:49.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.043+0000 7fe27a7fc700 1 -- 172.21.15.94:0/2040381055 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe268003570 con 0x7fe27c069990 2023-09-30T00:17:49.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.043+0000 7fe27b7fe700 1 -- 172.21.15.94:0/2040381055 >> v1:172.21.15.121:6789/0 conn(0x7fe27c0a49a0 legacy=0x7fe27c06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.043+0000 7fe27b7fe700 1 -- 172.21.15.94:0/2040381055 >> v1:172.21.15.94:6789/0 conn(0x7fe27c069990 legacy=0x7fe27c06e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.043+0000 7fe27b7fe700 1 -- 172.21.15.94:0/2040381055 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c06cd30 con 0x7fe27c0a5a50 2023-09-30T00:17:49.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.044+0000 7fe27b7fe700 1 --2- 172.21.15.94:0/2040381055 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 0x7fe27c0a5ea0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe274009e90 tx=0x7fe274009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5daa3b7f2fa8d808 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:49.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.044+0000 7fe27a7fc700 1 -- 172.21.15.94:0/2040381055 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27400e070 con 0x7fe27c0a5a50 2023-09-30T00:17:49.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.044+0000 7fe27a7fc700 1 -- 172.21.15.94:0/2040381055 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27400bc30 con 0x7fe27c0a5a50 2023-09-30T00:17:49.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.044+0000 7fe27a7fc700 1 -- 172.21.15.94:0/2040381055 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274012430 con 0x7fe27c0a5a50 2023-09-30T00:17:49.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.044+0000 7fe281716700 1 -- 172.21.15.94:0/2040381055 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 msgr2=0x7fe27c0a5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.044+0000 7fe281716700 1 --2- 172.21.15.94:0/2040381055 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 0x7fe27c0a5ea0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe274009e90 tx=0x7fe274009780 comp rx=0 tx=0).stop 2023-09-30T00:17:49.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.044+0000 7fe281716700 1 -- 172.21.15.94:0/2040381055 shutdown_connections 2023-09-30T00:17:49.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.044+0000 7fe281716700 1 --2- 172.21.15.94:0/2040381055 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 0x7fe27c0a5ea0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.044+0000 7fe281716700 1 -- 172.21.15.94:0/2040381055 >> 172.21.15.94:0/2040381055 conn(0x7fe27c0b3b10 msgr2=0x7fe27c0b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:49.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.044+0000 7fe281716700 1 -- 172.21.15.94:0/2040381055 shutdown_connections 2023-09-30T00:17:49.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.044+0000 7fe281716700 1 -- 172.21.15.94:0/2040381055 wait complete. 2023-09-30T00:17:49.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.045+0000 7fe281716700 1 Processor -- start 2023-09-30T00:17:49.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.045+0000 7fe281716700 1 -- start start 2023-09-30T00:17:49.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe281716700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe27c069990 0x7fe27c0972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe281716700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe27c0a49a0 0x7fe27c095f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe281716700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 0x7fe27c0a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe281716700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c097d40 con 0x7fe27c069990 2023-09-30T00:17:49.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe281716700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c096dc0 con 0x7fe27c0a49a0 2023-09-30T00:17:49.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe281716700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c097800 con 0x7fe27c0a5a50 2023-09-30T00:17:49.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe27b7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 0x7fe27c0a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe27affd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe27c069990 0x7fe27c0972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe27b7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 0x7fe27c0a0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51538/0 (socket says 172.21.15.94:51538) 2023-09-30T00:17:49.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe27b7fe700 1 -- 172.21.15.94:0/2600831720 learned_addr learned my addr 172.21.15.94:0/2600831720 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:49.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe27affd700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe27c069990 0x7fe27c0972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33422/0 (socket says 172.21.15.94:33422) 2023-09-30T00:17:49.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe2727fc700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe27c0a49a0 0x7fe27c095f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe27b7fe700 1 -- 172.21.15.94:0/2600831720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe27c0a49a0 msgr2=0x7fe27c095f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe27b7fe700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe27c0a49a0 0x7fe27c095f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe27b7fe700 1 -- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe27c069990 msgr2=0x7fe27c0972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe27b7fe700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe27c069990 0x7fe27c0972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe27b7fe700 1 -- 172.21.15.94:0/2600831720 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe274009a20 con 0x7fe27c0a5a50 2023-09-30T00:17:49.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe2727fc700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe27c0a49a0 0x7fe27c095f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:17:49.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe27affd700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe27c069990 0x7fe27c0972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:49.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe27b7fe700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 0x7fe27c0a0900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe274013000 tx=0x7fe274019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:49.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe2737fe700 1 -- 172.21.15.94:0/2600831720 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274012e80 con 0x7fe27c0a5a50 2023-09-30T00:17:49.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.046+0000 7fe2737fe700 1 -- 172.21.15.94:0/2600831720 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27401d630 con 0x7fe27c0a5a50 2023-09-30T00:17:49.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.047+0000 7fe281716700 1 -- 172.21.15.94:0/2600831720 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe27c097a80 con 0x7fe27c0a5a50 2023-09-30T00:17:49.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.047+0000 7fe2737fe700 1 -- 172.21.15.94:0/2600831720 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27401d9f0 con 0x7fe27c0a5a50 2023-09-30T00:17:49.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.047+0000 7fe281716700 1 -- 172.21.15.94:0/2600831720 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27c0a2190 con 0x7fe27c0a5a50 2023-09-30T00:17:49.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.047+0000 7fe281716700 1 -- 172.21.15.94:0/2600831720 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe27c14aa40 con 0x7fe27c0a5a50 2023-09-30T00:17:49.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.048+0000 7fe2737fe700 1 -- 172.21.15.94:0/2600831720 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe274024070 con 0x7fe27c0a5a50 2023-09-30T00:17:49.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.048+0000 7fe2737fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:49.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.049+0000 7fe2737fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:49.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.049+0000 7fe2737fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:49.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.049+0000 7fe2737fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:49.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.049+0000 7fe2737fe700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe25c098780 0x7fe25c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.049+0000 7fe27affd700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe25c098780 0x7fe25c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.049+0000 7fe2737fe700 1 -- 172.21.15.94:0/2600831720 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7fe2740c2ce0 con 0x7fe27c0a5a50 2023-09-30T00:17:49.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.049+0000 7fe27affd700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe25c098780 0x7fe25c09ac30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe26c007fd0 tx=0x7fe26c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:49.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.052+0000 7fe2737fe700 1 -- 172.21.15.94:0/2600831720 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe2740605a0 con 0x7fe27c0a5a50 2023-09-30T00:17:49.234 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.234+0000 7fe281716700 1 -- 172.21.15.94:0/2600831720 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe27c06fbd0 con 0x7fe27c0a5a50 2023-09-30T00:17:49.235 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.234+0000 7fe2737fe700 1 -- 172.21.15.94:0/2600831720 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe274094200 con 0x7fe27c0a5a50 2023-09-30T00:17:49.235 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 8 2023-09-30T00:17:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 10 mgrc shutdown 2023-09-30T00:17:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 1 -- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe25c098780 msgr2=0x7fe25c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe25c098780 0x7fe25c09ac30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe26c007fd0 tx=0x7fe26c009040 comp rx=0 tx=0).stop 2023-09-30T00:17:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 1 -- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 msgr2=0x7fe27c0a0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 0x7fe27c0a0900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe274013000 tx=0x7fe274019000 comp rx=0 tx=0).stop 2023-09-30T00:17:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 1 -- 172.21.15.94:0/2600831720 shutdown_connections 2023-09-30T00:17:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe25c098780 0x7fe25c09ac30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe27c0a5a50 0x7fe27c0a0900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.238 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe27c069990 0x7fe27c0972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 1 --2- 172.21.15.94:0/2600831720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe27c0a49a0 0x7fe27c095f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 1 -- 172.21.15.94:0/2600831720 >> 172.21.15.94:0/2600831720 conn(0x7fe27c0b3b10 msgr2=0x7fe27c13c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 1 -- 172.21.15.94:0/2600831720 shutdown_connections 2023-09-30T00:17:49.239 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.236+0000 7fe281716700 1 -- 172.21.15.94:0/2600831720 wait complete. 2023-09-30T00:17:49.248 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-30T00:17:49.365 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.365+0000 7f83936d3700 1 Processor -- start 2023-09-30T00:17:49.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.365+0000 7f83936d3700 1 -- start start 2023-09-30T00:17:49.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f83936d3700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f838c0a8bb0 0x7f838c0a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f83936d3700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f838c0a06c0 0x7f838c13fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f83936d3700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838c0681e0 con 0x7f838c067e10 2023-09-30T00:17:49.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f83936d3700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f838c0a0b00 con 0x7f838c0a8bb0 2023-09-30T00:17:49.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f83936d3700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f838c0a0d80 con 0x7f838c0a06c0 2023-09-30T00:17:49.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8390c6e700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f838c0a8bb0 0x7f838c0a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8390c6e700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f838c0a8bb0 0x7f838c0a92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33430/0 (socket says 172.21.15.94:33430) 2023-09-30T00:17:49.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8390c6e700 1 -- 172.21.15.94:0/3114687219 learned_addr learned my addr 172.21.15.94:0/3114687219 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:49.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f839146f700 1 --1- 172.21.15.94:0/3114687219 >> v1:172.21.15.121:6789/0 conn(0x7f838c067e10 0x7f838c0a24e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.94:51460/0 (socket says 172.21.15.94:51460) 2023-09-30T00:17:49.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8391c70700 1 --2- 172.21.15.94:0/3114687219 >> v2:172.21.15.121:3300/0 conn(0x7f838c0a06c0 0x7f838c13fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8390c6e700 1 -- 172.21.15.94:0/3114687219 >> v2:172.21.15.121:3300/0 conn(0x7f838c0a06c0 msgr2=0x7f838c13fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:49.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8390c6e700 1 --2- 172.21.15.94:0/3114687219 >> v2:172.21.15.121:3300/0 conn(0x7f838c0a06c0 0x7f838c13fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8390c6e700 1 -- 172.21.15.94:0/3114687219 >> v1:172.21.15.121:6789/0 conn(0x7f838c067e10 legacy=0x7f838c0a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8383fff700 1 -- 172.21.15.94:0/3114687219 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524482701 0 0) 0x7f838c0681e0 con 0x7f838c067e10 2023-09-30T00:17:49.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8390c6e700 1 -- 172.21.15.94:0/3114687219 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c0a1040 con 0x7f838c0a8bb0 2023-09-30T00:17:49.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8390c6e700 1 --2- 172.21.15.94:0/3114687219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f838c0a8bb0 0x7f838c0a92e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f838c0a6f50 tx=0x7f837c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc37be4a2eb7329 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:49.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8383fff700 1 -- 172.21.15.94:0/3114687219 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c009e70 con 0x7f838c0a8bb0 2023-09-30T00:17:49.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8383fff700 1 -- 172.21.15.94:0/3114687219 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837c00b860 con 0x7f838c0a8bb0 2023-09-30T00:17:49.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.366+0000 7f8383fff700 1 -- 172.21.15.94:0/3114687219 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c00bc00 con 0x7f838c0a8bb0 2023-09-30T00:17:49.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.367+0000 7f83936d3700 1 -- 172.21.15.94:0/3114687219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f838c0a8bb0 msgr2=0x7f838c0a92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.367+0000 7f83936d3700 1 --2- 172.21.15.94:0/3114687219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f838c0a8bb0 0x7f838c0a92e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f838c0a6f50 tx=0x7f837c009250 comp rx=0 tx=0).stop 2023-09-30T00:17:49.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.367+0000 7f83936d3700 1 -- 172.21.15.94:0/3114687219 shutdown_connections 2023-09-30T00:17:49.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.367+0000 7f83936d3700 1 --2- 172.21.15.94:0/3114687219 >> v2:172.21.15.121:3300/0 conn(0x7f838c0a06c0 0x7f838c13fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.370 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.367+0000 7f83936d3700 1 --2- 172.21.15.94:0/3114687219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f838c0a8bb0 0x7f838c0a92e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.371 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.367+0000 7f83936d3700 1 -- 172.21.15.94:0/3114687219 >> 172.21.15.94:0/3114687219 conn(0x7f838c06a270 msgr2=0x7f838c0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:49.371 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.367+0000 7f83936d3700 1 -- 172.21.15.94:0/3114687219 shutdown_connections 2023-09-30T00:17:49.371 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.367+0000 7f83936d3700 1 -- 172.21.15.94:0/3114687219 wait complete. 2023-09-30T00:17:49.371 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.368+0000 7f83936d3700 1 Processor -- start 2023-09-30T00:17:49.371 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.368+0000 7f83936d3700 1 -- start start 2023-09-30T00:17:49.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f83936d3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f838c067e10 0x7f838c153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f83936d3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f838c0a06c0 0x7f838c0af350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.373 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f83936d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f838c0a8bb0 0x7f838c0add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f83936d3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c1541e0 con 0x7f838c067e10 2023-09-30T00:17:49.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f83936d3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c0ae390 con 0x7f838c0a8bb0 2023-09-30T00:17:49.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f83936d3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838c0ae650 con 0x7f838c0a06c0 2023-09-30T00:17:49.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f839146f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f838c067e10 0x7f838c153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f839146f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f838c067e10 0x7f838c153c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33440/0 (socket says 172.21.15.94:33440) 2023-09-30T00:17:49.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f839146f700 1 -- 172.21.15.94:0/248281517 learned_addr learned my addr 172.21.15.94:0/248281517 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:49.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f8390c6e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f838c0a06c0 0x7f838c0af350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f8391c70700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f838c0a8bb0 0x7f838c0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f839146f700 1 -- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f838c0a06c0 msgr2=0x7f838c0af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f839146f700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f838c0a06c0 0x7f838c0af350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.374 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f839146f700 1 -- 172.21.15.94:0/248281517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f838c0a8bb0 msgr2=0x7f838c0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f839146f700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f838c0a8bb0 0x7f838c0add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f839146f700 1 -- 172.21.15.94:0/248281517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c0094f0 con 0x7f838c067e10 2023-09-30T00:17:49.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f8390c6e700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f838c0a06c0 0x7f838c0af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:49.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f8391c70700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f838c0a8bb0 0x7f838c0add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:17:49.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.369+0000 7f839146f700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f838c067e10 0x7f838c153c70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f838800cdd0 tx=0x7f8388001bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:49.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.370+0000 7f83817fa700 1 -- 172.21.15.94:0/248281517 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388005df0 con 0x7f838c067e10 2023-09-30T00:17:49.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.370+0000 7f83817fa700 1 -- 172.21.15.94:0/248281517 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838800a900 con 0x7f838c067e10 2023-09-30T00:17:49.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.370+0000 7f83936d3700 1 -- 172.21.15.94:0/248281517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838c0b0b30 con 0x7f838c067e10 2023-09-30T00:17:49.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.370+0000 7f83817fa700 1 -- 172.21.15.94:0/248281517 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838800acc0 con 0x7f838c067e10 2023-09-30T00:17:49.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.370+0000 7f83936d3700 1 -- 172.21.15.94:0/248281517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838c154670 con 0x7f838c067e10 2023-09-30T00:17:49.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.371+0000 7f83936d3700 1 -- 172.21.15.94:0/248281517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838c071630 con 0x7f838c067e10 2023-09-30T00:17:49.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.371+0000 7f83817fa700 1 -- 172.21.15.94:0/248281517 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f8388007300 con 0x7f838c067e10 2023-09-30T00:17:49.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.371+0000 7f83817fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:49.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.372+0000 7f83817fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:49.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.372+0000 7f83817fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:49.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.372+0000 7f83817fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:49.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.372+0000 7f83817fa700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8370098830 0x7f837009ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.372+0000 7f8390c6e700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8370098830 0x7f837009ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.372+0000 7f83817fa700 1 -- 172.21.15.94:0/248281517 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f83880c0320 con 0x7f838c067e10 2023-09-30T00:17:49.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.373+0000 7f8390c6e700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8370098830 0x7f837009ace0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f838c0a6f50 tx=0x7f837c017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:49.377 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.376+0000 7f83817fa700 1 -- 172.21.15.94:0/248281517 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f838808dc80 con 0x7f838c067e10 2023-09-30T00:17:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.556+0000 7f83936d3700 1 -- 172.21.15.94:0/248281517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f838c140340 con 0x7f838c067e10 2023-09-30T00:17:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.557+0000 7f83817fa700 1 -- 172.21.15.94:0/248281517 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) v1 ==== 93+0+10 (secure 0 0 0) 0x7f83880912a0 con 0x7f838c067e10 2023-09-30T00:17:49.557 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2023-09-30T00:17:49.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 10 mgrc shutdown 2023-09-30T00:17:49.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 1 -- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8370098830 msgr2=0x7f837009ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8370098830 0x7f837009ace0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f838c0a6f50 tx=0x7f837c017000 comp rx=0 tx=0).stop 2023-09-30T00:17:49.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 1 -- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f838c067e10 msgr2=0x7f838c153c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f838c067e10 0x7f838c153c70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f838800cdd0 tx=0x7f8388001bc0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 1 -- 172.21.15.94:0/248281517 shutdown_connections 2023-09-30T00:17:49.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8370098830 0x7f837009ace0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f838c0a8bb0 0x7f838c0add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f838c067e10 0x7f838c153c70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 1 --2- 172.21.15.94:0/248281517 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f838c0a06c0 0x7f838c0af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 1 -- 172.21.15.94:0/248281517 >> 172.21.15.94:0/248281517 conn(0x7f838c06a270 msgr2=0x7f838c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:49.561 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 1 -- 172.21.15.94:0/248281517 shutdown_connections 2023-09-30T00:17:49.561 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.559+0000 7f83936d3700 1 -- 172.21.15.94:0/248281517 wait complete. 2023-09-30T00:17:49.572 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-09-30T00:17:49.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.683+0000 7fba0b221700 1 Processor -- start 2023-09-30T00:17:49.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.683+0000 7fba0b221700 1 -- start start 2023-09-30T00:17:49.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fba0b221700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fba0406a460 0x7fba040b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fba0b221700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fba040b5680 0x7fba0414b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fba0b221700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba04069760 con 0x7fba040a46c0 2023-09-30T00:17:49.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fba0b221700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba040a57f0 con 0x7fba0406a460 2023-09-30T00:17:49.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fba0b221700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba040a4a90 con 0x7fba040b5680 2023-09-30T00:17:49.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fb9fbfff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fba0406a460 0x7fba040b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fba08fbd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fba040a46c0 0x7fba04068c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:32884/0 (socket says 172.21.15.94:32884) 2023-09-30T00:17:49.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fb9fbfff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fba0406a460 0x7fba040b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33450/0 (socket says 172.21.15.94:33450) 2023-09-30T00:17:49.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fba08fbd700 1 -- 172.21.15.94:0/1681489754 learned_addr learned my addr 172.21.15.94:0/1681489754 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:49.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fba097be700 1 --2- 172.21.15.94:0/1681489754 >> v2:172.21.15.121:3300/0 conn(0x7fba040b5680 0x7fba0414b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fb9fbfff700 1 -- 172.21.15.94:0/1681489754 >> v2:172.21.15.121:3300/0 conn(0x7fba040b5680 msgr2=0x7fba0414b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:49.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fb9fbfff700 1 --2- 172.21.15.94:0/1681489754 >> v2:172.21.15.121:3300/0 conn(0x7fba040b5680 0x7fba0414b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fb9fbfff700 1 -- 172.21.15.94:0/1681489754 >> v1:172.21.15.94:6789/0 conn(0x7fba040a46c0 legacy=0x7fba04068c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fb9fbfff700 1 -- 172.21.15.94:0/1681489754 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba04069c80 con 0x7fba0406a460 2023-09-30T00:17:49.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fb9fb7fe700 1 -- 172.21.15.94:0/1681489754 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568621153 0 0) 0x7fba04069760 con 0x7fba040a46c0 2023-09-30T00:17:49.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fba097be700 1 --2- 172.21.15.94:0/1681489754 >> v2:172.21.15.121:3300/0 conn(0x7fba040b5680 0x7fba0414b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:17:49.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.684+0000 7fb9fbfff700 1 --2- 172.21.15.94:0/1681489754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba0406a460 0x7fba040b5140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40098e0 tx=0x7fb9f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8b27bd7f9244e4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.685+0000 7fb9fb7fe700 1 -- 172.21.15.94:0/1681489754 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4009e80 con 0x7fba0406a460 2023-09-30T00:17:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.685+0000 7fb9fb7fe700 1 -- 172.21.15.94:0/1681489754 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f400b820 con 0x7fba0406a460 2023-09-30T00:17:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.685+0000 7fb9fb7fe700 1 -- 172.21.15.94:0/1681489754 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f400bba0 con 0x7fba0406a460 2023-09-30T00:17:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.685+0000 7fba0b221700 1 -- 172.21.15.94:0/1681489754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba0406a460 msgr2=0x7fba040b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.685+0000 7fba0b221700 1 --2- 172.21.15.94:0/1681489754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba0406a460 0x7fba040b5140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40098e0 tx=0x7fb9f40091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.685+0000 7fba0b221700 1 -- 172.21.15.94:0/1681489754 shutdown_connections 2023-09-30T00:17:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.685+0000 7fba0b221700 1 --2- 172.21.15.94:0/1681489754 >> v2:172.21.15.121:3300/0 conn(0x7fba040b5680 0x7fba0414b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.685+0000 7fba0b221700 1 --2- 172.21.15.94:0/1681489754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba0406a460 0x7fba040b5140 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.685+0000 7fba0b221700 1 -- 172.21.15.94:0/1681489754 >> 172.21.15.94:0/1681489754 conn(0x7fba0406cbf0 msgr2=0x7fba0406cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.685+0000 7fba0b221700 1 -- 172.21.15.94:0/1681489754 shutdown_connections 2023-09-30T00:17:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.685+0000 7fba0b221700 1 -- 172.21.15.94:0/1681489754 wait complete. 2023-09-30T00:17:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.686+0000 7fba0b221700 1 Processor -- start 2023-09-30T00:17:49.692 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.686+0000 7fba0b221700 1 -- start start 2023-09-30T00:17:49.692 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fba0b221700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba0406a460 0x7fba040b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.692 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fba0b221700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fba040a46c0 0x7fba0411d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.692 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fba0b221700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba040b5680 0x7fba040acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.692 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fba0b221700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0413fc80 con 0x7fba040b5680 2023-09-30T00:17:49.692 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fba0b221700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba041394f0 con 0x7fba0406a460 2023-09-30T00:17:49.692 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fba0b221700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba040ae240 con 0x7fba040a46c0 2023-09-30T00:17:49.692 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fb9fbfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fba040a46c0 0x7fba0411d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.692 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fb9fbfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fba040a46c0 0x7fba0411d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51560/0 (socket says 172.21.15.94:51560) 2023-09-30T00:17:49.692 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fb9fbfff700 1 -- 172.21.15.94:0/10173341 learned_addr learned my addr 172.21.15.94:0/10173341 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:49.692 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fba097be700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba040b5680 0x7fba040acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fba08fbd700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba0406a460 0x7fba040b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fb9fbfff700 1 -- 172.21.15.94:0/10173341 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba0406a460 msgr2=0x7fba040b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fb9fbfff700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba0406a460 0x7fba040b0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fb9fbfff700 1 -- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba040b5680 msgr2=0x7fba040acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fb9fbfff700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba040b5680 0x7fba040acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fb9fbfff700 1 -- 172.21.15.94:0/10173341 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f4009420 con 0x7fba040a46c0 2023-09-30T00:17:49.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fba097be700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba040b5680 0x7fba040acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:49.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fb9fbfff700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fba040a46c0 0x7fba0411d230 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400b3c0 tx=0x7fb9f4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:49.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fb9f8ff9700 1 -- 172.21.15.94:0/10173341 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f400bba0 con 0x7fba040a46c0 2023-09-30T00:17:49.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fb9f8ff9700 1 -- 172.21.15.94:0/10173341 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f4000c90 con 0x7fba040a46c0 2023-09-30T00:17:49.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fba0b221700 1 -- 172.21.15.94:0/10173341 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0411d770 con 0x7fba040a46c0 2023-09-30T00:17:49.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.687+0000 7fb9f8ff9700 1 -- 172.21.15.94:0/10173341 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4011650 con 0x7fba040a46c0 2023-09-30T00:17:49.694 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.688+0000 7fba0b221700 1 -- 172.21.15.94:0/10173341 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0411dbd0 con 0x7fba040a46c0 2023-09-30T00:17:49.694 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.688+0000 7fba0b221700 1 -- 172.21.15.94:0/10173341 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0406fbd0 con 0x7fba040a46c0 2023-09-30T00:17:49.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.689+0000 7fb9f8ff9700 1 -- 172.21.15.94:0/10173341 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb9f40119c0 con 0x7fba040a46c0 2023-09-30T00:17:49.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.689+0000 7fb9f8ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:49.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.690+0000 7fb9f8ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:49.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.690+0000 7fb9f8ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:49.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.690+0000 7fb9f8ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:49.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.690+0000 7fb9f8ff9700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb9f0098780 0x7fb9f009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:49.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.690+0000 7fba08fbd700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb9f0098780 0x7fb9f009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:49.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.690+0000 7fb9f8ff9700 1 -- 172.21.15.94:0/10173341 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7fb9f40c1a20 con 0x7fba040a46c0 2023-09-30T00:17:49.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.690+0000 7fba08fbd700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb9f0098780 0x7fb9f009ac30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fba00007fd0 tx=0x7fba00007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:49.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.694+0000 7fb9f8ff9700 1 -- 172.21.15.94:0/10173341 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb9f408f4b0 con 0x7fba040a46c0 2023-09-30T00:17:49.879 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.878+0000 7fba0b221700 1 -- 172.21.15.94:0/10173341 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fba0411d900 con 0x7fba040a46c0 2023-09-30T00:17:49.879 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.878+0000 7fb9f8ff9700 1 -- 172.21.15.94:0/10173341 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v15) v1 ==== 84+0+126 (secure 0 0 0) 0x7fb9f4015090 con 0x7fba040a46c0 2023-09-30T00:17:49.879 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:49.879 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2023-09-30T00:17:49.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2023-09-30T00:17:49.879 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2023-09-30T00:17:49.880 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": "rbd" 2023-09-30T00:17:49.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2023-09-30T00:17:49.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2023-09-30T00:17:49.880 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 2, 2023-09-30T00:17:49.880 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2023-09-30T00:17:49.880 INFO:teuthology.orchestra.run.smithi094.stdout: } 2023-09-30T00:17:49.880 INFO:teuthology.orchestra.run.smithi094.stdout:] 2023-09-30T00:17:49.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.880+0000 7fba0b221700 10 mgrc shutdown 2023-09-30T00:17:49.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.880+0000 7fba0b221700 1 -- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb9f0098780 msgr2=0x7fb9f009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.882 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.880+0000 7fba0b221700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb9f0098780 0x7fb9f009ac30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fba00007fd0 tx=0x7fba00007d30 comp rx=0 tx=0).stop 2023-09-30T00:17:49.882 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.880+0000 7fba0b221700 1 -- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fba040a46c0 msgr2=0x7fba0411d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:49.882 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.880+0000 7fba0b221700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fba040a46c0 0x7fba0411d230 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400b3c0 tx=0x7fb9f4017000 comp rx=0 tx=0).stop 2023-09-30T00:17:49.882 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.880+0000 7fba0b221700 1 -- 172.21.15.94:0/10173341 shutdown_connections 2023-09-30T00:17:49.882 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.880+0000 7fba0b221700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb9f0098780 0x7fb9f009ac30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.882 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.880+0000 7fba0b221700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba040b5680 0x7fba040acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.882 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.880+0000 7fba0b221700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba0406a460 0x7fba040b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.882 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.880+0000 7fba0b221700 1 --2- 172.21.15.94:0/10173341 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fba040a46c0 0x7fba0411d230 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:49.882 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.880+0000 7fba0b221700 1 -- 172.21.15.94:0/10173341 >> 172.21.15.94:0/10173341 conn(0x7fba0406cbf0 msgr2=0x7fba0406cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:49.882 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.880+0000 7fba0b221700 1 -- 172.21.15.94:0/10173341 shutdown_connections 2023-09-30T00:17:49.883 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:49.881+0000 7fba0b221700 1 -- 172.21.15.94:0/10173341 wait complete. 2023-09-30T00:17:49.892 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-09-30T00:17:49.893 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-09-30T00:17:50.009 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.009+0000 7fe1f0e55700 1 Processor -- start 2023-09-30T00:17:50.010 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.009+0000 7fe1f0e55700 1 -- start start 2023-09-30T00:17:50.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1f0e55700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe1ec13ff50 0x7fe1ec140360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1f0e55700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1ec0688f0 con 0x7fe1ec068520 2023-09-30T00:17:50.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1f0e55700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1ec0a2990 con 0x7fe1ec0a9240 2023-09-30T00:17:50.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1f0e55700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1ec0a8d00 con 0x7fe1ec13ff50 2023-09-30T00:17:50.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1eb7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe1ec068520 0x7fe1ec0a2170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:32898/0 (socket says 172.21.15.94:32898) 2023-09-30T00:17:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1eb7fe700 1 -- 172.21.15.94:0/1649085507 learned_addr learned my addr 172.21.15.94:0/1649085507 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ebfff700 1 --2- 172.21.15.94:0/1649085507 >> v2:172.21.15.121:3300/0 conn(0x7fe1ec13ff50 0x7fe1ec140360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628204242 0 0) 0x7fe1ec0688f0 con 0x7fe1ec068520 2023-09-30T00:17:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d4003570 con 0x7fe1ec068520 2023-09-30T00:17:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402958036 0 0) 0x7fe1ec0a2990 con 0x7fe1ec0a9240 2023-09-30T00:17:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1ec0688f0 con 0x7fe1ec0a9240 2023-09-30T00:17:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 100039000 0 0) 0x7fe1d4003570 con 0x7fe1ec068520 2023-09-30T00:17:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 >> v2:172.21.15.121:3300/0 conn(0x7fe1ec13ff50 msgr2=0x7fe1ec140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 --2- 172.21.15.94:0/1649085507 >> v2:172.21.15.121:3300/0 conn(0x7fe1ec13ff50 0x7fe1ec140360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 >> v1:172.21.15.121:6789/0 conn(0x7fe1ec0a9240 legacy=0x7fe1ec0a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec0a1420 con 0x7fe1ec068520 2023-09-30T00:17:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3153706020 0 0) 0x7fe1dc0015f0 con 0x7fe1ec068520 2023-09-30T00:17:50.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ebfff700 1 --2- 172.21.15.94:0/1649085507 >> v2:172.21.15.121:3300/0 conn(0x7fe1ec13ff50 0x7fe1ec140360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:50.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 >> v1:172.21.15.94:6789/0 conn(0x7fe1ec068520 legacy=0x7fe1ec0a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:50.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 --2- 172.21.15.94:0/1649085507 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe1d4003c40 0x7fe1d4004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 --2- 172.21.15.94:0/1649085507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 0x7fe1d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 --2- 172.21.15.94:0/1649085507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1d4005210 0x7fe1d400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec0a8d00 con 0x7fe1d4004810 2023-09-30T00:17:50.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d400c130 con 0x7fe1d4005210 2023-09-30T00:17:50.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d400c2c0 con 0x7fe1d4003c40 2023-09-30T00:17:50.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1eb7fe700 1 --2- 172.21.15.94:0/1649085507 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe1d4003c40 0x7fe1d4004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.010+0000 7fe1eaffd700 1 --2- 172.21.15.94:0/1649085507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 0x7fe1d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1ebfff700 1 --2- 172.21.15.94:0/1649085507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1d4005210 0x7fe1d400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1eaffd700 1 -- 172.21.15.94:0/1649085507 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe1d4003c40 msgr2=0x7fe1d4004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:50.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1eaffd700 1 --2- 172.21.15.94:0/1649085507 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe1d4003c40 0x7fe1d4004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1eaffd700 1 -- 172.21.15.94:0/1649085507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1d4005210 msgr2=0x7fe1d400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:50.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1eaffd700 1 --2- 172.21.15.94:0/1649085507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1d4005210 0x7fe1d400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1ebfff700 1 -- 172.21.15.94:0/1649085507 reap_dead start 2023-09-30T00:17:50.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1eaffd700 1 -- 172.21.15.94:0/1649085507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d4003570 con 0x7fe1d4004810 2023-09-30T00:17:50.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1eaffd700 1 --2- 172.21.15.94:0/1649085507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 0x7fe1d4004c80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0001a40 tx=0x7fe1e0007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a64197899fa0dc4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:50.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e000c520 con 0x7fe1d4004810 2023-09-30T00:17:50.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e000c680 con 0x7fe1d4004810 2023-09-30T00:17:50.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1ea7fc700 1 -- 172.21.15.94:0/1649085507 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e000ca20 con 0x7fe1d4004810 2023-09-30T00:17:50.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1f0e55700 1 -- 172.21.15.94:0/1649085507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 msgr2=0x7fe1d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:50.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1f0e55700 1 --2- 172.21.15.94:0/1649085507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 0x7fe1d4004c80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0001a40 tx=0x7fe1e0007d30 comp rx=0 tx=0).stop 2023-09-30T00:17:50.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1f0e55700 1 -- 172.21.15.94:0/1649085507 shutdown_connections 2023-09-30T00:17:50.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1f0e55700 1 --2- 172.21.15.94:0/1649085507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 0x7fe1d4004c80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.011+0000 7fe1f0e55700 1 -- 172.21.15.94:0/1649085507 >> 172.21.15.94:0/1649085507 conn(0x7fe1ec070dc0 msgr2=0x7fe1ec067bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:50.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.012+0000 7fe1f0e55700 1 -- 172.21.15.94:0/1649085507 shutdown_connections 2023-09-30T00:17:50.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.012+0000 7fe1f0e55700 1 -- 172.21.15.94:0/1649085507 wait complete. 2023-09-30T00:17:50.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1f0e55700 1 Processor -- start 2023-09-30T00:17:50.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1f0e55700 1 -- start start 2023-09-30T00:17:50.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1f0e55700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 0x7fe1ec139550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1f0e55700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1ec0b68d0 0x7fe1ec0b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1f0e55700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe1ec09f6b0 0x7fe1ec09fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1f0e55700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec0a0060 con 0x7fe1d4004810 2023-09-30T00:17:50.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1f0e55700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec144660 con 0x7fe1ec0b68d0 2023-09-30T00:17:50.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1f0e55700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec144cb0 con 0x7fe1ec09f6b0 2023-09-30T00:17:50.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1eb7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 0x7fe1ec139550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1eb7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 0x7fe1ec139550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33470/0 (socket says 172.21.15.94:33470) 2023-09-30T00:17:50.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1ebfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe1ec09f6b0 0x7fe1ec09fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1eb7fe700 1 -- 172.21.15.94:0/569985565 learned_addr learned my addr 172.21.15.94:0/569985565 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:50.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1eaffd700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1ec0b68d0 0x7fe1ec0b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1eb7fe700 1 -- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe1ec09f6b0 msgr2=0x7fe1ec09fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:50.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1eb7fe700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe1ec09f6b0 0x7fe1ec09fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1eb7fe700 1 -- 172.21.15.94:0/569985565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1ec0b68d0 msgr2=0x7fe1ec0b4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:50.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1eb7fe700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1ec0b68d0 0x7fe1ec0b4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1eb7fe700 1 -- 172.21.15.94:0/569985565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec0b6de0 con 0x7fe1d4004810 2023-09-30T00:17:50.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.013+0000 7fe1ebfff700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe1ec09f6b0 0x7fe1ec09fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:50.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.014+0000 7fe1eb7fe700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 0x7fe1ec139550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00f8b0 tx=0x7fe1dc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:50.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.014+0000 7fe1d37fe700 1 -- 172.21.15.94:0/569985565 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc0072e0 con 0x7fe1d4004810 2023-09-30T00:17:50.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.014+0000 7fe1d37fe700 1 -- 172.21.15.94:0/569985565 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1dc007440 con 0x7fe1d4004810 2023-09-30T00:17:50.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.014+0000 7fe1d37fe700 1 -- 172.21.15.94:0/569985565 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc0057f0 con 0x7fe1d4004810 2023-09-30T00:17:50.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.014+0000 7fe1f0e55700 1 -- 172.21.15.94:0/569985565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1ec0b70a0 con 0x7fe1d4004810 2023-09-30T00:17:50.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.014+0000 7fe1f0e55700 1 -- 172.21.15.94:0/569985565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ec0b5370 con 0x7fe1d4004810 2023-09-30T00:17:50.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.015+0000 7fe1f0e55700 1 -- 172.21.15.94:0/569985565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1ec0692d0 con 0x7fe1d4004810 2023-09-30T00:17:50.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.016+0000 7fe1d37fe700 1 -- 172.21.15.94:0/569985565 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe1dc005b80 con 0x7fe1d4004810 2023-09-30T00:17:50.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.016+0000 7fe1d37fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:50.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.016+0000 7fe1d37fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:50.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.016+0000 7fe1d37fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:50.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.016+0000 7fe1d37fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:50.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.016+0000 7fe1d37fe700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe1d80987b0 0x7fe1d809ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.016+0000 7fe1eaffd700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe1d80987b0 0x7fe1d809ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.016+0000 7fe1d37fe700 1 -- 172.21.15.94:0/569985565 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7fe1dc0bff50 con 0x7fe1d4004810 2023-09-30T00:17:50.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.017+0000 7fe1eaffd700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe1d80987b0 0x7fe1d809ac60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec0a1000 tx=0x7fe1e0015000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:50.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.020+0000 7fe1d37fe700 1 -- 172.21.15.94:0/569985565 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe1dc08d9e0 con 0x7fe1d4004810 2023-09-30T00:17:50.203 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.202+0000 7fe1f0e55700 1 -- 172.21.15.94:0/569985565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe1ec04c380 con 0x7fe1d4004810 2023-09-30T00:17:50.566 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.566+0000 7fe1d37fe700 1 -- 172.21.15.94:0/569985565 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v16) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe1dc091000 con 0x7fe1d4004810 2023-09-30T00:17:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 10 mgrc shutdown 2023-09-30T00:17:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 1 -- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe1d80987b0 msgr2=0x7fe1d809ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe1d80987b0 0x7fe1d809ac60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec0a1000 tx=0x7fe1e0015000 comp rx=0 tx=0).stop 2023-09-30T00:17:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 1 -- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 msgr2=0x7fe1ec139550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 0x7fe1ec139550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00f8b0 tx=0x7fe1dc009040 comp rx=0 tx=0).stop 2023-09-30T00:17:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 1 -- 172.21.15.94:0/569985565 shutdown_connections 2023-09-30T00:17:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe1d80987b0 0x7fe1d809ac60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe1ec09f6b0 0x7fe1ec09fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1d4004810 0x7fe1ec139550 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 1 --2- 172.21.15.94:0/569985565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1ec0b68d0 0x7fe1ec0b4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 1 -- 172.21.15.94:0/569985565 >> 172.21.15.94:0/569985565 conn(0x7fe1ec070dc0 msgr2=0x7fe1ec0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 1 -- 172.21.15.94:0/569985565 shutdown_connections 2023-09-30T00:17:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.568+0000 7fe1f0e55700 1 -- 172.21.15.94:0/569985565 wait complete. 2023-09-30T00:17:50.570 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2023-09-30T00:17:50.581 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-09-30T00:17:50.702 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.702+0000 7f0145b38700 1 Processor -- start 2023-09-30T00:17:50.702 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.702+0000 7f0145b38700 1 -- start start 2023-09-30T00:17:50.704 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.702+0000 7f0145b38700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01400a3190 0x7f01400b7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.702+0000 7f0145b38700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f01400a2300 0x7f014014b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.702+0000 7f0145b38700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01400b7420 con 0x7f0140068570 2023-09-30T00:17:50.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.702+0000 7f0145b38700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01401408a0 con 0x7f01400a3190 2023-09-30T00:17:50.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.702+0000 7f0145b38700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01400a2730 con 0x7f01400a2300 2023-09-30T00:17:50.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.702+0000 7f013effd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01400a3190 0x7f01400b7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.702+0000 7f013effd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01400a3190 0x7f01400b7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33472/0 (socket says 172.21.15.94:33472) 2023-09-30T00:17:50.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.702+0000 7f013effd700 1 -- 172.21.15.94:0/3876332709 learned_addr learned my addr 172.21.15.94:0/3876332709 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:50.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f013ffff700 1 --2- 172.21.15.94:0/3876332709 >> v2:172.21.15.121:3300/0 conn(0x7f01400a2300 0x7f014014b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f013effd700 1 -- 172.21.15.94:0/3876332709 >> v2:172.21.15.121:3300/0 conn(0x7f01400a2300 msgr2=0x7f014014b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:50.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f013effd700 1 --2- 172.21.15.94:0/3876332709 >> v2:172.21.15.121:3300/0 conn(0x7f01400a2300 0x7f014014b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f013effd700 1 -- 172.21.15.94:0/3876332709 >> v1:172.21.15.121:6789/0 conn(0x7f0140068570 legacy=0x7f0140140190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f013effd700 1 -- 172.21.15.94:0/3876332709 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01400a29c0 con 0x7f01400a3190 2023-09-30T00:17:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f013effd700 1 --2- 172.21.15.94:0/3876332709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01400a3190 0x7f01400b7850 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0140141ac0 tx=0x7f0134009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef3e91642c94c00 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f013e7fc700 1 -- 172.21.15.94:0/3876332709 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0134009e70 con 0x7f01400a3190 2023-09-30T00:17:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f013e7fc700 1 -- 172.21.15.94:0/3876332709 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013400b860 con 0x7f01400a3190 2023-09-30T00:17:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f013e7fc700 1 -- 172.21.15.94:0/3876332709 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013400bc00 con 0x7f01400a3190 2023-09-30T00:17:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f0145b38700 1 -- 172.21.15.94:0/3876332709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01400a3190 msgr2=0x7f01400b7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f0145b38700 1 --2- 172.21.15.94:0/3876332709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01400a3190 0x7f01400b7850 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0140141ac0 tx=0x7f0134009250 comp rx=0 tx=0).stop 2023-09-30T00:17:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f0145b38700 1 -- 172.21.15.94:0/3876332709 shutdown_connections 2023-09-30T00:17:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f0145b38700 1 --2- 172.21.15.94:0/3876332709 >> v2:172.21.15.121:3300/0 conn(0x7f01400a2300 0x7f014014b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f0145b38700 1 --2- 172.21.15.94:0/3876332709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01400a3190 0x7f01400b7850 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.703+0000 7f0145b38700 1 -- 172.21.15.94:0/3876332709 >> 172.21.15.94:0/3876332709 conn(0x7f01400a9810 msgr2=0x7f0140067ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.704+0000 7f0145b38700 1 -- 172.21.15.94:0/3876332709 shutdown_connections 2023-09-30T00:17:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.704+0000 7f0145b38700 1 -- 172.21.15.94:0/3876332709 wait complete. 2023-09-30T00:17:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f0145b38700 1 Processor -- start 2023-09-30T00:17:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f0145b38700 1 -- start start 2023-09-30T00:17:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f0145b38700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0140068570 0x7f014013e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f0145b38700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01400a2300 0x7f01400a8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f0145b38700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01400a3190 0x7f01400a44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f0145b38700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014013bd60 con 0x7f01400a3190 2023-09-30T00:17:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f0145b38700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01400a84a0 con 0x7f0140068570 2023-09-30T00:17:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f0145b38700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014013bef0 con 0x7f01400a2300 2023-09-30T00:17:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f013ffff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01400a3190 0x7f01400a44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f013effd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01400a2300 0x7f01400a8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f013ffff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01400a3190 0x7f01400a44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33476/0 (socket says 172.21.15.94:33476) 2023-09-30T00:17:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f013ffff700 1 -- 172.21.15.94:0/541756459 learned_addr learned my addr 172.21.15.94:0/541756459 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.705+0000 7f013f7fe700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0140068570 0x7f014013e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.706+0000 7f013effd700 1 -- 172.21.15.94:0/541756459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0140068570 msgr2=0x7f014013e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:50.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.706+0000 7f013effd700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0140068570 0x7f014013e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.706+0000 7f013effd700 1 -- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01400a3190 msgr2=0x7f01400a44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.706+0000 7f013effd700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01400a3190 0x7f01400a44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.706+0000 7f013effd700 1 -- 172.21.15.94:0/541756459 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01340094f0 con 0x7f01400a2300 2023-09-30T00:17:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.706+0000 7f013ffff700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01400a3190 0x7f01400a44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.706+0000 7f013effd700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01400a2300 0x7f01400a8ab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0140069aa0 tx=0x7f0134017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.706+0000 7f01277fe700 1 -- 172.21.15.94:0/541756459 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013400bbe0 con 0x7f01400a2300 2023-09-30T00:17:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.706+0000 7f01277fe700 1 -- 172.21.15.94:0/541756459 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0134000c90 con 0x7f01400a2300 2023-09-30T00:17:50.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.706+0000 7f0145b38700 1 -- 172.21.15.94:0/541756459 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01400a4b80 con 0x7f01400a2300 2023-09-30T00:17:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.706+0000 7f01277fe700 1 -- 172.21.15.94:0/541756459 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0134011650 con 0x7f01400a2300 2023-09-30T00:17:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.706+0000 7f0145b38700 1 -- 172.21.15.94:0/541756459 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01400a4e40 con 0x7f01400a2300 2023-09-30T00:17:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.707+0000 7f0145b38700 1 -- 172.21.15.94:0/541756459 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01400a6cc0 con 0x7f01400a2300 2023-09-30T00:17:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.708+0000 7f01277fe700 1 -- 172.21.15.94:0/541756459 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f0134011a80 con 0x7f01400a2300 2023-09-30T00:17:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.708+0000 7f01277fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.708+0000 7f01277fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.708+0000 7f01277fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.708+0000 7f01277fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.708+0000 7f01277fe700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f012c098780 0x7f012c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.708+0000 7f013f7fe700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f012c098780 0x7f012c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:50.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.708+0000 7f01277fe700 1 -- 172.21.15.94:0/541756459 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3848+0+0 (secure 0 0 0) 0x7f01340c1960 con 0x7f01400a2300 2023-09-30T00:17:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.709+0000 7f013f7fe700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f012c098780 0x7f012c09ac30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0130003a20 tx=0x7f0130007f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:50.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.712+0000 7f01277fe700 1 -- 172.21.15.94:0/541756459 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f01400a6cc0 con 0x7f01400a2300 2023-09-30T00:17:50.894 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:50.893+0000 7f0145b38700 1 -- 172.21.15.94:0/541756459 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f01400693b0 con 0x7f01400a2300 2023-09-30T00:17:51.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.569+0000 7f01277fe700 1 -- 172.21.15.94:0/541756459 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v17) v1 ==== 170+0+0 (secure 0 0 0) 0x7f01340150d0 con 0x7f01400a2300 2023-09-30T00:17:51.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 10 mgrc shutdown 2023-09-30T00:17:51.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 1 -- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f012c098780 msgr2=0x7f012c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:51.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f012c098780 0x7f012c09ac30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0130003a20 tx=0x7f0130007f10 comp rx=0 tx=0).stop 2023-09-30T00:17:51.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 1 -- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01400a2300 msgr2=0x7f01400a8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:51.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01400a2300 0x7f01400a8ab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0140069aa0 tx=0x7f0134017000 comp rx=0 tx=0).stop 2023-09-30T00:17:51.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 1 -- 172.21.15.94:0/541756459 shutdown_connections 2023-09-30T00:17:51.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f012c098780 0x7f012c09ac30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:51.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01400a3190 0x7f01400a44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:51.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0140068570 0x7f014013e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:51.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 1 --2- 172.21.15.94:0/541756459 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01400a2300 0x7f01400a8ab0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:51.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 1 -- 172.21.15.94:0/541756459 >> 172.21.15.94:0/541756459 conn(0x7f01400a9810 msgr2=0x7f01400b5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:51.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 1 -- 172.21.15.94:0/541756459 shutdown_connections 2023-09-30T00:17:51.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.572+0000 7f0145b38700 1 -- 172.21.15.94:0/541756459 wait complete. 2023-09-30T00:17:51.575 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2023-09-30T00:17:51.588 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-09-30T00:17:51.703 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.703+0000 7f5c93deb700 1 Processor -- start 2023-09-30T00:17:51.704 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.703+0000 7f5c93deb700 1 -- start start 2023-09-30T00:17:51.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.703+0000 7f5c93deb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 0x7f5c8c0a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:51.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.703+0000 7f5c93deb700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c0a0b40 con 0x7f5c8c0a06c0 2023-09-30T00:17:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c93deb700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c0681e0 con 0x7f5c8c0a8bb0 2023-09-30T00:17:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c93deb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c0a0d80 con 0x7f5c8c067e10 2023-09-30T00:17:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c91b87700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 0x7f5c8c0a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c92388700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5c8c0a06c0 0x7f5c8c13fe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:32904/0 (socket says 172.21.15.94:32904) 2023-09-30T00:17:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c91b87700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 0x7f5c8c0a92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51596/0 (socket says 172.21.15.94:51596) 2023-09-30T00:17:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c92388700 1 -- 172.21.15.94:0/2852026985 learned_addr learned my addr 172.21.15.94:0/2852026985 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c90b85700 1 -- 172.21.15.94:0/2852026985 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666691690 0 0) 0x7f5c84001490 con 0x7f5c8c0a06c0 2023-09-30T00:17:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c90b85700 1 -- 172.21.15.94:0/2852026985 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c78003570 con 0x7f5c8c0a06c0 2023-09-30T00:17:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c91b87700 1 -- 172.21.15.94:0/2852026985 >> v1:172.21.15.121:6789/0 conn(0x7f5c8c0a8bb0 legacy=0x7f5c8c0a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c91b87700 1 -- 172.21.15.94:0/2852026985 >> v1:172.21.15.94:6789/0 conn(0x7f5c8c0a06c0 legacy=0x7f5c8c13fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c91b87700 1 -- 172.21.15.94:0/2852026985 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c0a1040 con 0x7f5c8c067e10 2023-09-30T00:17:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c91b87700 1 --2- 172.21.15.94:0/2852026985 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 0x7f5c8c0a92e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009910 tx=0x7f5c880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=560871dab0b55927 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c90b85700 1 -- 172.21.15.94:0/2852026985 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8800b680 con 0x7f5c8c067e10 2023-09-30T00:17:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c90b85700 1 -- 172.21.15.94:0/2852026985 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8800b7e0 con 0x7f5c8c067e10 2023-09-30T00:17:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.704+0000 7f5c90b85700 1 -- 172.21.15.94:0/2852026985 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8800bba0 con 0x7f5c8c067e10 2023-09-30T00:17:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.705+0000 7f5c93deb700 1 -- 172.21.15.94:0/2852026985 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 msgr2=0x7f5c8c0a92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.705+0000 7f5c93deb700 1 --2- 172.21.15.94:0/2852026985 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 0x7f5c8c0a92e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009910 tx=0x7f5c880091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.705+0000 7f5c93deb700 1 -- 172.21.15.94:0/2852026985 shutdown_connections 2023-09-30T00:17:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.705+0000 7f5c93deb700 1 --2- 172.21.15.94:0/2852026985 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 0x7f5c8c0a92e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.705+0000 7f5c93deb700 1 -- 172.21.15.94:0/2852026985 >> 172.21.15.94:0/2852026985 conn(0x7f5c8c06a270 msgr2=0x7f5c8c0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.705+0000 7f5c93deb700 1 -- 172.21.15.94:0/2852026985 shutdown_connections 2023-09-30T00:17:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.705+0000 7f5c93deb700 1 -- 172.21.15.94:0/2852026985 wait complete. 2023-09-30T00:17:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.706+0000 7f5c93deb700 1 Processor -- start 2023-09-30T00:17:51.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.706+0000 7f5c93deb700 1 -- start start 2023-09-30T00:17:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.706+0000 7f5c93deb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 0x7f5c8c13ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.706+0000 7f5c93deb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c8c0a06c0 0x7f5c8c138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.706+0000 7f5c93deb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c0a8bb0 0x7f5c8c13d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.706+0000 7f5c93deb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c13efd0 con 0x7f5c8c0a8bb0 2023-09-30T00:17:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.706+0000 7f5c93deb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c1446e0 con 0x7f5c8c0a06c0 2023-09-30T00:17:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.706+0000 7f5c93deb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c0a48e0 con 0x7f5c8c067e10 2023-09-30T00:17:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c91b87700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 0x7f5c8c13ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c92388700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c0a8bb0 0x7f5c8c13d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c92388700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c0a8bb0 0x7f5c8c13d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33480/0 (socket says 172.21.15.94:33480) 2023-09-30T00:17:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c91b87700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 0x7f5c8c13ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51610/0 (socket says 172.21.15.94:51610) 2023-09-30T00:17:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c92388700 1 -- 172.21.15.94:0/2656466776 learned_addr learned my addr 172.21.15.94:0/2656466776 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c91386700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c8c0a06c0 0x7f5c8c138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c91b87700 1 -- 172.21.15.94:0/2656466776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c8c0a06c0 msgr2=0x7f5c8c138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c91b87700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c8c0a06c0 0x7f5c8c138fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:51.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c91b87700 1 -- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c0a8bb0 msgr2=0x7f5c8c13d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:51.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c91b87700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c0a8bb0 0x7f5c8c13d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:51.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c91b87700 1 -- 172.21.15.94:0/2656466776 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c88009420 con 0x7f5c8c067e10 2023-09-30T00:17:51.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c92388700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c0a8bb0 0x7f5c8c13d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:51.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c91b87700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 0x7f5c8c13ea90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5c88016000 tx=0x7f5c88017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:51.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c81ffb700 1 -- 172.21.15.94:0/2656466776 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8800bb60 con 0x7f5c8c067e10 2023-09-30T00:17:51.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c81ffb700 1 -- 172.21.15.94:0/2656466776 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c88000c90 con 0x7f5c8c067e10 2023-09-30T00:17:51.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c81ffb700 1 -- 172.21.15.94:0/2656466776 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8801a660 con 0x7f5c8c067e10 2023-09-30T00:17:51.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c93deb700 1 -- 172.21.15.94:0/2656466776 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c0a4ba0 con 0x7f5c8c067e10 2023-09-30T00:17:51.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.707+0000 7f5c93deb700 1 -- 172.21.15.94:0/2656466776 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c0a6330 con 0x7f5c8c067e10 2023-09-30T00:17:51.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.708+0000 7f5c93deb700 1 -- 172.21.15.94:0/2656466776 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c0a9600 con 0x7f5c8c067e10 2023-09-30T00:17:51.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.709+0000 7f5c81ffb700 1 -- 172.21.15.94:0/2656466776 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5c8801aa10 con 0x7f5c8c067e10 2023-09-30T00:17:51.714 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.709+0000 7f5c81ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:51.714 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.710+0000 7f5c81ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:51.714 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.710+0000 7f5c81ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:51.714 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.710+0000 7f5c81ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:51.714 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.710+0000 7f5c81ffb700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5c70098780 0x7f5c7009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:51.714 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.710+0000 7f5c91386700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5c70098780 0x7f5c7009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:51.714 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.710+0000 7f5c81ffb700 1 -- 172.21.15.94:0/2656466776 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4230+0+0 (secure 0 0 0) 0x7f5c880c2260 con 0x7f5c8c067e10 2023-09-30T00:17:51.714 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.710+0000 7f5c91386700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5c70098780 0x7f5c7009ac30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c144ce0 tx=0x7f5c7c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:51.714 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.713+0000 7f5c81ffb700 1 -- 172.21.15.94:0/2656466776 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5c8808fa00 con 0x7f5c8c067e10 2023-09-30T00:17:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.935+0000 7f5c93deb700 1 -- 172.21.15.94:0/2656466776 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5c8c068650 con 0x7f5c8c067e10 2023-09-30T00:17:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.941+0000 7f5c81ffb700 1 -- 172.21.15.94:0/2656466776 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f5c880150c0 con 0x7f5c8c067e10 2023-09-30T00:17:51.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 10 mgrc shutdown 2023-09-30T00:17:51.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 1 -- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5c70098780 msgr2=0x7f5c7009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:51.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5c70098780 0x7f5c7009ac30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c144ce0 tx=0x7f5c7c009040 comp rx=0 tx=0).stop 2023-09-30T00:17:51.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 1 -- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 msgr2=0x7f5c8c13ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:51.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 0x7f5c8c13ea90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5c88016000 tx=0x7f5c88017000 comp rx=0 tx=0).stop 2023-09-30T00:17:51.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 1 -- 172.21.15.94:0/2656466776 shutdown_connections 2023-09-30T00:17:51.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5c70098780 0x7f5c7009ac30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:51.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c8c067e10 0x7f5c8c13ea90 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5c88016000 tx=0x7f5c88017000 comp rx=0 tx=0).stop 2023-09-30T00:17:51.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c0a8bb0 0x7f5c8c13d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:51.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 1 --2- 172.21.15.94:0/2656466776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c8c0a06c0 0x7f5c8c138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:51.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 1 -- 172.21.15.94:0/2656466776 >> 172.21.15.94:0/2656466776 conn(0x7f5c8c06a270 msgr2=0x7f5c8c0b5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:51.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 1 -- 172.21.15.94:0/2656466776 shutdown_connections 2023-09-30T00:17:51.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:51.944+0000 7f5c93deb700 1 -- 172.21.15.94:0/2656466776 wait complete. 2023-09-30T00:17:51.946 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 3 and data pool 4 2023-09-30T00:17:51.959 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:17:52.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.073+0000 7f2d03382700 1 Processor -- start 2023-09-30T00:17:52.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.073+0000 7f2d03382700 1 -- start start 2023-09-30T00:17:52.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.074+0000 7f2d03382700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 0x7f2cfc142550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.074+0000 7f2d03382700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc067c40 con 0x7f2cfc134490 2023-09-30T00:17:52.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.074+0000 7f2d03382700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc142a90 con 0x7f2cfc067870 2023-09-30T00:17:52.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.074+0000 7f2d03382700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc06a620 con 0x7f2cfc142160 2023-09-30T00:17:52.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.074+0000 7f2d0091d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 0x7f2cfc142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.074+0000 7f2d0091d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 0x7f2cfc142550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51626/0 (socket says 172.21.15.94:51626) 2023-09-30T00:17:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.074+0000 7f2d0191f700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2cfc134490 0x7f2cfc134910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:32908/0 (socket says 172.21.15.94:32908) 2023-09-30T00:17:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.074+0000 7f2d0091d700 1 -- 172.21.15.94:0/3447279397 learned_addr learned my addr 172.21.15.94:0/3447279397 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.074+0000 7f2cf3fff700 1 -- 172.21.15.94:0/3447279397 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249928385 0 0) 0x7f2cfc067c40 con 0x7f2cfc134490 2023-09-30T00:17:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.074+0000 7f2d0091d700 1 -- 172.21.15.94:0/3447279397 >> v1:172.21.15.121:6789/0 conn(0x7f2cfc067870 legacy=0x7f2cfc09d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.074+0000 7f2d0091d700 1 -- 172.21.15.94:0/3447279397 >> v1:172.21.15.94:6789/0 conn(0x7f2cfc134490 legacy=0x7f2cfc134910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.074+0000 7f2d0091d700 1 -- 172.21.15.94:0/3447279397 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc0a4110 con 0x7f2cfc142160 2023-09-30T00:17:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.075+0000 7f2d0091d700 1 --2- 172.21.15.94:0/3447279397 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 0x7f2cfc142550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0099e0 tx=0x7f2cec009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=678e5e7c4982764f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.075+0000 7f2cf3fff700 1 -- 172.21.15.94:0/3447279397 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec00b860 con 0x7f2cfc142160 2023-09-30T00:17:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.075+0000 7f2cf3fff700 1 -- 172.21.15.94:0/3447279397 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec00b9c0 con 0x7f2cfc142160 2023-09-30T00:17:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.075+0000 7f2cf3fff700 1 -- 172.21.15.94:0/3447279397 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec00bd60 con 0x7f2cfc142160 2023-09-30T00:17:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.075+0000 7f2d03382700 1 -- 172.21.15.94:0/3447279397 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 msgr2=0x7f2cfc142550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.075+0000 7f2d03382700 1 --2- 172.21.15.94:0/3447279397 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 0x7f2cfc142550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0099e0 tx=0x7f2cec009250 comp rx=0 tx=0).stop 2023-09-30T00:17:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.075+0000 7f2d03382700 1 -- 172.21.15.94:0/3447279397 shutdown_connections 2023-09-30T00:17:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.075+0000 7f2d03382700 1 --2- 172.21.15.94:0/3447279397 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 0x7f2cfc142550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.075+0000 7f2d03382700 1 -- 172.21.15.94:0/3447279397 >> 172.21.15.94:0/3447279397 conn(0x7f2cfc06ba30 msgr2=0x7f2cfc06be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.075+0000 7f2d03382700 1 -- 172.21.15.94:0/3447279397 shutdown_connections 2023-09-30T00:17:52.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.075+0000 7f2d03382700 1 -- 172.21.15.94:0/3447279397 wait complete. 2023-09-30T00:17:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.076+0000 7f2d03382700 1 Processor -- start 2023-09-30T00:17:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.076+0000 7f2d03382700 1 -- start start 2023-09-30T00:17:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d03382700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cfc067870 0x7f2cfc1359e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d03382700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cfc134490 0x7f2cfc0a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d03382700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 0x7f2cfc0a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d03382700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc135f20 con 0x7f2cfc067870 2023-09-30T00:17:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d03382700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc137490 con 0x7f2cfc134490 2023-09-30T00:17:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d03382700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc136140 con 0x7f2cfc142160 2023-09-30T00:17:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d0111e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cfc067870 0x7f2cfc1359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d0111e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cfc067870 0x7f2cfc1359e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33494/0 (socket says 172.21.15.94:33494) 2023-09-30T00:17:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d0191f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 0x7f2cfc0a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d0111e700 1 -- 172.21.15.94:0/2003286419 learned_addr learned my addr 172.21.15.94:0/2003286419 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d0091d700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cfc134490 0x7f2cfc0a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d0191f700 1 -- 172.21.15.94:0/2003286419 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cfc134490 msgr2=0x7f2cfc0a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d0191f700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cfc134490 0x7f2cfc0a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d0191f700 1 -- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cfc067870 msgr2=0x7f2cfc1359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d0191f700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cfc067870 0x7f2cfc1359e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d0191f700 1 -- 172.21.15.94:0/2003286419 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec0094f0 con 0x7f2cfc142160 2023-09-30T00:17:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d0111e700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cfc067870 0x7f2cfc1359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.077+0000 7f2d0191f700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 0x7f2cfc0a1d70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400cca0 tx=0x7f2cf4001a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.078+0000 7f2cf17fa700 1 -- 172.21.15.94:0/2003286419 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4005df0 con 0x7f2cfc142160 2023-09-30T00:17:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.078+0000 7f2cf17fa700 1 -- 172.21.15.94:0/2003286419 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf400a620 con 0x7f2cfc142160 2023-09-30T00:17:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.078+0000 7f2d03382700 1 -- 172.21.15.94:0/2003286419 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc136460 con 0x7f2cfc142160 2023-09-30T00:17:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.078+0000 7f2cf17fa700 1 -- 172.21.15.94:0/2003286419 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf400a9e0 con 0x7f2cfc142160 2023-09-30T00:17:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.078+0000 7f2d03382700 1 -- 172.21.15.94:0/2003286419 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc0abac0 con 0x7f2cfc142160 2023-09-30T00:17:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.078+0000 7f2d03382700 1 -- 172.21.15.94:0/2003286419 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cfc1372b0 con 0x7f2cfc142160 2023-09-30T00:17:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.079+0000 7f2cf17fa700 1 -- 172.21.15.94:0/2003286419 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2cf40070f0 con 0x7f2cfc142160 2023-09-30T00:17:52.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.079+0000 7f2cf17fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.080+0000 7f2cf17fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.080+0000 7f2cf17fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.080+0000 7f2cf17fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.080+0000 7f2cf17fa700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2ce0098780 0x7f2ce009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.080+0000 7f2d0111e700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2ce0098780 0x7f2ce009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.080+0000 7f2cf17fa700 1 -- 172.21.15.94:0/2003286419 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f2cf40c0360 con 0x7f2cfc142160 2023-09-30T00:17:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.080+0000 7f2d0111e700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2ce0098780 0x7f2ce009ac30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8001e90 tx=0x7f2cf8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:52.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.084+0000 7f2cf17fa700 1 -- 172.21.15.94:0/2003286419 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2cf408daa0 con 0x7f2cfc142160 2023-09-30T00:17:52.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.270+0000 7f2d03382700 1 -- 172.21.15.94:0/2003286419 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cfc071630 con 0x7f2cfc142160 2023-09-30T00:17:52.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.271+0000 7f2cf17fa700 1 -- 172.21.15.94:0/2003286419 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12210 (secure 0 0 0) 0x7f2cf40910c0 con 0x7f2cfc142160 2023-09-30T00:17:52.271 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:52.272 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":18,"fsid":"3c891173-587e-4c6f-9c6a-04ec750eedad","created":"2023-09-30T00:17:34.347592+0000","modified":"2023-09-30T00:17:51.937194+0000","last_up_change":"2023-09-30T00:17:42.879328+0000","last_in_change":"2023-09-30T00:17:35.583427+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T00:17:43.481613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T00:17:45.892643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-09-30T00:17:50.220028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-09-30T00:17:50.911066+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c508b402-f8ce-4461-823b-9e35fcd8d40f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":84364},{"type":"v1","addr":"172.21.15.94:6801","nonce":84364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":84364},{"type":"v1","addr":"172.21.15.94:6803","nonce":84364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":84364},{"type":"v1","addr":"172.21.15.94:6807","nonce":84364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":84364},{"type":"v1","addr":"172.21.15.94:6805","nonce":84364}]},"public_addr":"172.21.15.94:6801/84364","cluster_addr":"172.21.15.94:6803/84364","heartbeat_back_addr":"172.21.15.94:6807/84364","heartbeat_front_addr":"172.21.15.94:6805/84364","state":["exists","up"]},{"osd":1,"uuid":"e96ecdf0-bb38-4eb1-be99-54f813c12888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":86850},{"type":"v1","addr":"172.21.15.121:6817","nonce":86850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":86850},{"type":"v1","addr":"172.21.15.121:6819","nonce":86850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":86850},{"type":"v1","addr":"172.21.15.121:6823","nonce":86850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":86850},{"type":"v1","addr":"172.21.15.121:6821","nonce":86850}]},"public_addr":"172.21.15.121:6817/86850","cluster_addr":"172.21.15.121:6819/86850","heartbeat_back_addr":"172.21.15.121:6823/86850","heartbeat_front_addr":"172.21.15.121:6821/86850","state":["exists","up"]},{"osd":2,"uuid":"68700ae4-764c-4dff-9cf2-5f33132cd0d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":86820},{"type":"v1","addr":"172.21.15.121:6801","nonce":86820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":86820},{"type":"v1","addr":"172.21.15.121:6803","nonce":86820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":86820},{"type":"v1","addr":"172.21.15.121:6810","nonce":86820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":86820},{"type":"v1","addr":"172.21.15.121:6806","nonce":86820}]},"public_addr":"172.21.15.121:6801/86820","cluster_addr":"172.21.15.121:6803/86820","heartbeat_back_addr":"172.21.15.121:6810/86820","heartbeat_front_addr":"172.21.15.121:6806/86820","state":["exists","up"]},{"osd":3,"uuid":"54f17ec8-2561-42e8-bbd7-262130ffdac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":86821},{"type":"v1","addr":"172.21.15.121:6807","nonce":86821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":86821},{"type":"v1","addr":"172.21.15.121:6811","nonce":86821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":86821},{"type":"v1","addr":"172.21.15.121:6815","nonce":86821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":86821},{"type":"v1","addr":"172.21.15.121:6813","nonce":86821}]},"public_addr":"172.21.15.121:6807/86821","cluster_addr":"172.21.15.121:6811/86821","heartbeat_back_addr":"172.21.15.121:6815/86821","heartbeat_front_addr":"172.21.15.121:6813/86821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:39.951865+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.769149+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.738994+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.748197+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:17:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 10 mgrc shutdown 2023-09-30T00:17:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 1 -- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2ce0098780 msgr2=0x7f2ce009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2ce0098780 0x7f2ce009ac30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8001e90 tx=0x7f2cf8009040 comp rx=0 tx=0).stop 2023-09-30T00:17:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 1 -- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 msgr2=0x7f2cfc0a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 0x7f2cfc0a1d70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400cca0 tx=0x7f2cf4001a40 comp rx=0 tx=0).stop 2023-09-30T00:17:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 1 -- 172.21.15.94:0/2003286419 shutdown_connections 2023-09-30T00:17:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2ce0098780 0x7f2ce009ac30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cfc142160 0x7f2cfc0a1d70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cfc067870 0x7f2cfc1359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 1 --2- 172.21.15.94:0/2003286419 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cfc134490 0x7f2cfc0a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 1 -- 172.21.15.94:0/2003286419 >> 172.21.15.94:0/2003286419 conn(0x7f2cfc06ba30 msgr2=0x7f2cfc06be00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:52.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 1 -- 172.21.15.94:0/2003286419 shutdown_connections 2023-09-30T00:17:52.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.273+0000 7f2d03382700 1 -- 172.21.15.94:0/2003286419 wait complete. 2023-09-30T00:17:52.285 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:17:52.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.399+0000 7f3a1e646700 1 Processor -- start 2023-09-30T00:17:52.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.399+0000 7f3a1e646700 1 -- start start 2023-09-30T00:17:52.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a1e646700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180b52b0 0x7f3a18141930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a1e646700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a18141e70 con 0x7f3a180a7b70 2023-09-30T00:17:52.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a1e646700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a180b5680 con 0x7f3a180a51a0 2023-09-30T00:17:52.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a1e646700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a180a5710 con 0x7f3a180b52b0 2023-09-30T00:17:52.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a1ce43700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3a180a7b70 0x7f3a181343f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:32922/0 (socket says 172.21.15.94:32922) 2023-09-30T00:17:52.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a1ce43700 1 -- 172.21.15.94:0/2050755165 learned_addr learned my addr 172.21.15.94:0/2050755165 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:52.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a1d644700 1 --2- 172.21.15.94:0/2050755165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180b52b0 0x7f3a18141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a0ffff700 1 -- 172.21.15.94:0/2050755165 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906103636 0 0) 0x7f3a08001110 con 0x7f3a180a7b70 2023-09-30T00:17:52.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a0ffff700 1 -- 172.21.15.94:0/2050755165 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a04003570 con 0x7f3a180a7b70 2023-09-30T00:17:52.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a1d644700 1 -- 172.21.15.94:0/2050755165 >> v1:172.21.15.121:6789/0 conn(0x7f3a180a51a0 legacy=0x7f3a180a59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a1d644700 1 -- 172.21.15.94:0/2050755165 >> v1:172.21.15.94:6789/0 conn(0x7f3a180a7b70 legacy=0x7f3a181343f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a1d644700 1 -- 172.21.15.94:0/2050755165 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a18134b00 con 0x7f3a180b52b0 2023-09-30T00:17:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a1d644700 1 --2- 172.21.15.94:0/2050755165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180b52b0 0x7f3a18141930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009e90 tx=0x7f3a10009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdecf58a4827fb9b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a0ffff700 1 -- 172.21.15.94:0/2050755165 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1000e070 con 0x7f3a180b52b0 2023-09-30T00:17:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a0ffff700 1 -- 172.21.15.94:0/2050755165 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1000bc30 con 0x7f3a180b52b0 2023-09-30T00:17:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.400+0000 7f3a0ffff700 1 -- 172.21.15.94:0/2050755165 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10012430 con 0x7f3a180b52b0 2023-09-30T00:17:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.401+0000 7f3a1e646700 1 -- 172.21.15.94:0/2050755165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180b52b0 msgr2=0x7f3a18141930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.401+0000 7f3a1e646700 1 --2- 172.21.15.94:0/2050755165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180b52b0 0x7f3a18141930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009e90 tx=0x7f3a10009780 comp rx=0 tx=0).stop 2023-09-30T00:17:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.401+0000 7f3a1e646700 1 -- 172.21.15.94:0/2050755165 shutdown_connections 2023-09-30T00:17:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.401+0000 7f3a1e646700 1 --2- 172.21.15.94:0/2050755165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180b52b0 0x7f3a18141930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.401+0000 7f3a1e646700 1 -- 172.21.15.94:0/2050755165 >> 172.21.15.94:0/2050755165 conn(0x7f3a1806c940 msgr2=0x7f3a1806cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:52.404 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.401+0000 7f3a1e646700 1 -- 172.21.15.94:0/2050755165 shutdown_connections 2023-09-30T00:17:52.404 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.401+0000 7f3a1e646700 1 -- 172.21.15.94:0/2050755165 wait complete. 2023-09-30T00:17:52.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.402+0000 7f3a1e646700 1 Processor -- start 2023-09-30T00:17:52.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.402+0000 7f3a1e646700 1 -- start start 2023-09-30T00:17:52.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.402+0000 7f3a1e646700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180a51a0 0x7f3a180aea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.402+0000 7f3a1e646700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a180a7b70 0x7f3a1814e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.402+0000 7f3a1e646700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a180b52b0 0x7f3a1814eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1e646700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1814f070 con 0x7f3a180a7b70 2023-09-30T00:17:52.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1e646700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a180a9b70 con 0x7f3a180b52b0 2023-09-30T00:17:52.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1e646700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a180ae310 con 0x7f3a180a51a0 2023-09-30T00:17:52.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1d644700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180a51a0 0x7f3a180aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1d644700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180a51a0 0x7f3a180aea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51650/0 (socket says 172.21.15.94:51650) 2023-09-30T00:17:52.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1d644700 1 -- 172.21.15.94:0/1367732994 learned_addr learned my addr 172.21.15.94:0/1367732994 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:52.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1ce43700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a180a7b70 0x7f3a1814e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1de45700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a180b52b0 0x7f3a1814eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1d644700 1 -- 172.21.15.94:0/1367732994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a180b52b0 msgr2=0x7f3a1814eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1d644700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a180b52b0 0x7f3a1814eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1d644700 1 -- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a180a7b70 msgr2=0x7f3a1814e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1d644700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a180a7b70 0x7f3a1814e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1d644700 1 -- 172.21.15.94:0/1367732994 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a10009a20 con 0x7f3a180a51a0 2023-09-30T00:17:52.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1ce43700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a180a7b70 0x7f3a1814e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:52.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1d644700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180a51a0 0x7f3a180aea80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3a10013000 tx=0x7f3a10019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:52.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a0d7fa700 1 -- 172.21.15.94:0/1367732994 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10001090 con 0x7f3a180a51a0 2023-09-30T00:17:52.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a0d7fa700 1 -- 172.21.15.94:0/1367732994 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a10012e60 con 0x7f3a180a51a0 2023-09-30T00:17:52.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1e646700 1 -- 172.21.15.94:0/1367732994 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a1814f380 con 0x7f3a180a51a0 2023-09-30T00:17:52.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a0d7fa700 1 -- 172.21.15.94:0/1367732994 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a100276a0 con 0x7f3a180a51a0 2023-09-30T00:17:52.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.403+0000 7f3a1e646700 1 -- 172.21.15.94:0/1367732994 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a180ae030 con 0x7f3a180a51a0 2023-09-30T00:17:52.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.404+0000 7f3a1e646700 1 -- 172.21.15.94:0/1367732994 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a1804c380 con 0x7f3a180a51a0 2023-09-30T00:17:52.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.405+0000 7f3a0d7fa700 1 -- 172.21.15.94:0/1367732994 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f3a10024070 con 0x7f3a180a51a0 2023-09-30T00:17:52.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.405+0000 7f3a0d7fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:52.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.405+0000 7f3a0d7fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:52.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.405+0000 7f3a0d7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:52.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.405+0000 7f3a0d7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:52.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.405+0000 7f3a0d7fa700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f39fc098700 0x7f39fc09abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.406+0000 7f3a1ce43700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f39fc098700 0x7f39fc09abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.406+0000 7f3a0d7fa700 1 -- 172.21.15.94:0/1367732994 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f3a100c2850 con 0x7f3a180a51a0 2023-09-30T00:17:52.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.406+0000 7f3a1ce43700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f39fc098700 0x7f39fc09abb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3a18133ed0 tx=0x7f3a08007bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:52.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.410+0000 7f3a0d7fa700 1 -- 172.21.15.94:0/1367732994 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a1008ff90 con 0x7f3a180a51a0 2023-09-30T00:17:52.595 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.595+0000 7f3a1e646700 1 -- 172.21.15.94:0/1367732994 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a1814f320 con 0x7f3a180a51a0 2023-09-30T00:17:52.596 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.595+0000 7f3a0d7fa700 1 -- 172.21.15.94:0/1367732994 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12210 (secure 0 0 0) 0x7f3a1002c020 con 0x7f3a180a51a0 2023-09-30T00:17:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:52.596 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":18,"fsid":"3c891173-587e-4c6f-9c6a-04ec750eedad","created":"2023-09-30T00:17:34.347592+0000","modified":"2023-09-30T00:17:51.937194+0000","last_up_change":"2023-09-30T00:17:42.879328+0000","last_in_change":"2023-09-30T00:17:35.583427+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T00:17:43.481613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T00:17:45.892643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-09-30T00:17:50.220028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-09-30T00:17:50.911066+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c508b402-f8ce-4461-823b-9e35fcd8d40f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":84364},{"type":"v1","addr":"172.21.15.94:6801","nonce":84364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":84364},{"type":"v1","addr":"172.21.15.94:6803","nonce":84364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":84364},{"type":"v1","addr":"172.21.15.94:6807","nonce":84364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":84364},{"type":"v1","addr":"172.21.15.94:6805","nonce":84364}]},"public_addr":"172.21.15.94:6801/84364","cluster_addr":"172.21.15.94:6803/84364","heartbeat_back_addr":"172.21.15.94:6807/84364","heartbeat_front_addr":"172.21.15.94:6805/84364","state":["exists","up"]},{"osd":1,"uuid":"e96ecdf0-bb38-4eb1-be99-54f813c12888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":86850},{"type":"v1","addr":"172.21.15.121:6817","nonce":86850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":86850},{"type":"v1","addr":"172.21.15.121:6819","nonce":86850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":86850},{"type":"v1","addr":"172.21.15.121:6823","nonce":86850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":86850},{"type":"v1","addr":"172.21.15.121:6821","nonce":86850}]},"public_addr":"172.21.15.121:6817/86850","cluster_addr":"172.21.15.121:6819/86850","heartbeat_back_addr":"172.21.15.121:6823/86850","heartbeat_front_addr":"172.21.15.121:6821/86850","state":["exists","up"]},{"osd":2,"uuid":"68700ae4-764c-4dff-9cf2-5f33132cd0d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":86820},{"type":"v1","addr":"172.21.15.121:6801","nonce":86820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":86820},{"type":"v1","addr":"172.21.15.121:6803","nonce":86820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":86820},{"type":"v1","addr":"172.21.15.121:6810","nonce":86820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":86820},{"type":"v1","addr":"172.21.15.121:6806","nonce":86820}]},"public_addr":"172.21.15.121:6801/86820","cluster_addr":"172.21.15.121:6803/86820","heartbeat_back_addr":"172.21.15.121:6810/86820","heartbeat_front_addr":"172.21.15.121:6806/86820","state":["exists","up"]},{"osd":3,"uuid":"54f17ec8-2561-42e8-bbd7-262130ffdac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":86821},{"type":"v1","addr":"172.21.15.121:6807","nonce":86821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":86821},{"type":"v1","addr":"172.21.15.121:6811","nonce":86821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":86821},{"type":"v1","addr":"172.21.15.121:6815","nonce":86821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":86821},{"type":"v1","addr":"172.21.15.121:6813","nonce":86821}]},"public_addr":"172.21.15.121:6807/86821","cluster_addr":"172.21.15.121:6811/86821","heartbeat_back_addr":"172.21.15.121:6815/86821","heartbeat_front_addr":"172.21.15.121:6813/86821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:39.951865+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.769149+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.738994+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.748197+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:17:52.598 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 10 mgrc shutdown 2023-09-30T00:17:52.599 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 1 -- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f39fc098700 msgr2=0x7f39fc09abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.599 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f39fc098700 0x7f39fc09abb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3a18133ed0 tx=0x7f3a08007bf0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.599 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 1 -- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180a51a0 msgr2=0x7f3a180aea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.599 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180a51a0 0x7f3a180aea80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3a10013000 tx=0x7f3a10019000 comp rx=0 tx=0).stop 2023-09-30T00:17:52.599 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 1 -- 172.21.15.94:0/1367732994 shutdown_connections 2023-09-30T00:17:52.599 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f39fc098700 0x7f39fc09abb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.599 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a180a51a0 0x7f3a180aea80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.599 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a180b52b0 0x7f3a1814eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.599 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 1 --2- 172.21.15.94:0/1367732994 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a180a7b70 0x7f3a1814e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.599 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 1 -- 172.21.15.94:0/1367732994 >> 172.21.15.94:0/1367732994 conn(0x7f3a1806c940 msgr2=0x7f3a1806d890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:52.599 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 1 -- 172.21.15.94:0/1367732994 shutdown_connections 2023-09-30T00:17:52.600 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.598+0000 7f3a1e646700 1 -- 172.21.15.94:0/1367732994 wait complete. 2023-09-30T00:17:52.610 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-09-30T00:17:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.725+0000 7f6392f8b700 1 Processor -- start 2023-09-30T00:17:52.726 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.725+0000 7f6392f8b700 1 -- start start 2023-09-30T00:17:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.725+0000 7f6392f8b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f638c067e10 0x7f638c0a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.728 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6392f8b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c0a06c0 0x7f638c13fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6392f8b700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c0a0b40 con 0x7f638c0a8bb0 2023-09-30T00:17:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6392f8b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638c0681e0 con 0x7f638c067e10 2023-09-30T00:17:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6392f8b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638c0a0d80 con 0x7f638c0a06c0 2023-09-30T00:17:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6390d27700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f638c067e10 0x7f638c0a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6391528700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c0a06c0 0x7f638c13fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6390d27700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f638c067e10 0x7f638c0a92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33520/0 (socket says 172.21.15.94:33520) 2023-09-30T00:17:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6391528700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c0a06c0 0x7f638c13fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51654/0 (socket says 172.21.15.94:51654) 2023-09-30T00:17:52.729 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6390d27700 1 -- 172.21.15.94:0/2493911280 learned_addr learned my addr 172.21.15.94:0/2493911280 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6390d27700 1 -- 172.21.15.94:0/2493911280 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c0a06c0 msgr2=0x7f638c13fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6390d27700 1 --2- 172.21.15.94:0/2493911280 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c0a06c0 0x7f638c13fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6390d27700 1 -- 172.21.15.94:0/2493911280 >> v1:172.21.15.94:6789/0 conn(0x7f638c0a8bb0 legacy=0x7f638c0a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6390d27700 1 -- 172.21.15.94:0/2493911280 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c0a1040 con 0x7f638c067e10 2023-09-30T00:17:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f638b7fe700 1 -- 172.21.15.94:0/2493911280 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793264824 0 0) 0x7f6378001110 con 0x7f638c0a8bb0 2023-09-30T00:17:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6391528700 1 --2- 172.21.15.94:0/2493911280 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c0a06c0 0x7f638c13fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f6390d27700 1 --2- 172.21.15.94:0/2493911280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f638c067e10 0x7f638c0a92e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f63800099b0 tx=0x7f6380009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75acda1a77945c1c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f638b7fe700 1 -- 172.21.15.94:0/2493911280 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6380009ea0 con 0x7f638c067e10 2023-09-30T00:17:52.730 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f638b7fe700 1 -- 172.21.15.94:0/2493911280 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638000b860 con 0x7f638c067e10 2023-09-30T00:17:52.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.726+0000 7f638b7fe700 1 -- 172.21.15.94:0/2493911280 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638000bc00 con 0x7f638c067e10 2023-09-30T00:17:52.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.727+0000 7f6392f8b700 1 -- 172.21.15.94:0/2493911280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f638c067e10 msgr2=0x7f638c0a92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.727+0000 7f6392f8b700 1 --2- 172.21.15.94:0/2493911280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f638c067e10 0x7f638c0a92e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f63800099b0 tx=0x7f6380009250 comp rx=0 tx=0).stop 2023-09-30T00:17:52.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.727+0000 7f6392f8b700 1 -- 172.21.15.94:0/2493911280 shutdown_connections 2023-09-30T00:17:52.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.727+0000 7f6392f8b700 1 --2- 172.21.15.94:0/2493911280 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c0a06c0 0x7f638c13fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.727+0000 7f6392f8b700 1 --2- 172.21.15.94:0/2493911280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f638c067e10 0x7f638c0a92e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.727+0000 7f6392f8b700 1 -- 172.21.15.94:0/2493911280 >> 172.21.15.94:0/2493911280 conn(0x7f638c06a270 msgr2=0x7f638c0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:52.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.727+0000 7f6392f8b700 1 -- 172.21.15.94:0/2493911280 shutdown_connections 2023-09-30T00:17:52.731 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.727+0000 7f6392f8b700 1 -- 172.21.15.94:0/2493911280 wait complete. 2023-09-30T00:17:52.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.728+0000 7f6392f8b700 1 Processor -- start 2023-09-30T00:17:52.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.728+0000 7f6392f8b700 1 -- start start 2023-09-30T00:17:52.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.728+0000 7f6392f8b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c067e10 0x7f638c0a5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6392f8b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f638c0a06c0 0x7f638c13a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6392f8b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f638c0a8bb0 0x7f638c1385f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6392f8b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c1502f0 con 0x7f638c0a8bb0 2023-09-30T00:17:52.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6392f8b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c0ab5f0 con 0x7f638c0a06c0 2023-09-30T00:17:52.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6392f8b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638c138b30 con 0x7f638c067e10 2023-09-30T00:17:52.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6390d27700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c067e10 0x7f638c0a5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6391528700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f638c0a8bb0 0x7f638c1385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6390d27700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c067e10 0x7f638c0a5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51670/0 (socket says 172.21.15.94:51670) 2023-09-30T00:17:52.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6391528700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f638c0a8bb0 0x7f638c1385f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33522/0 (socket says 172.21.15.94:33522) 2023-09-30T00:17:52.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6390d27700 1 -- 172.21.15.94:0/4114896268 learned_addr learned my addr 172.21.15.94:0/4114896268 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:52.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f638bfff700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f638c0a06c0 0x7f638c13a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6390d27700 1 -- 172.21.15.94:0/4114896268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f638c0a06c0 msgr2=0x7f638c13a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6390d27700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f638c0a06c0 0x7f638c13a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6390d27700 1 -- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f638c0a8bb0 msgr2=0x7f638c1385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:52.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6390d27700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f638c0a8bb0 0x7f638c1385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:52.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6390d27700 1 -- 172.21.15.94:0/4114896268 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6384009a20 con 0x7f638c067e10 2023-09-30T00:17:52.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6391528700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f638c0a8bb0 0x7f638c1385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:52.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6390d27700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c067e10 0x7f638c0a5fe0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6380006fd0 tx=0x7f6380017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:52.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6388ff9700 1 -- 172.21.15.94:0/4114896268 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638000bbe0 con 0x7f638c067e10 2023-09-30T00:17:52.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6388ff9700 1 -- 172.21.15.94:0/4114896268 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6380011420 con 0x7f638c067e10 2023-09-30T00:17:52.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6392f8b700 1 -- 172.21.15.94:0/4114896268 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63800094f0 con 0x7f638c067e10 2023-09-30T00:17:52.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6388ff9700 1 -- 172.21.15.94:0/4114896268 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63800117c0 con 0x7f638c067e10 2023-09-30T00:17:52.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.729+0000 7f6392f8b700 1 -- 172.21.15.94:0/4114896268 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638c0aaba0 con 0x7f638c067e10 2023-09-30T00:17:52.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.730+0000 7f6392f8b700 1 -- 172.21.15.94:0/4114896268 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638c071630 con 0x7f638c067e10 2023-09-30T00:17:52.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.731+0000 7f6388ff9700 1 -- 172.21.15.94:0/4114896268 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6380011b70 con 0x7f638c067e10 2023-09-30T00:17:52.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.731+0000 7f6388ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:52.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.732+0000 7f6388ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:52.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.732+0000 7f6388ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:52.738 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.732+0000 7f6388ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:52.738 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.732+0000 7f6388ff9700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6370098780 0x7f637009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:52.738 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.732+0000 7f638bfff700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6370098780 0x7f637009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:52.738 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.732+0000 7f6388ff9700 1 -- 172.21.15.94:0/4114896268 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f63800c2200 con 0x7f638c067e10 2023-09-30T00:17:52.738 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.732+0000 7f638bfff700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6370098780 0x7f637009ac30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f638c13f1f0 tx=0x7f6378009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:52.738 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.735+0000 7f6388ff9700 1 -- 172.21.15.94:0/4114896268 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f638008f940 con 0x7f638c067e10 2023-09-30T00:17:52.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:52.962+0000 7f6392f8b700 1 -- 172.21.15.94:0/4114896268 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f638c140340 con 0x7f638c067e10 2023-09-30T00:17:53.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.945+0000 7f6388ff9700 1 -- 172.21.15.94:0/4114896268 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6380092f60 con 0x7f638c067e10 2023-09-30T00:17:53.948 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.947+0000 7f6392f8b700 10 mgrc shutdown 2023-09-30T00:17:53.948 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.947+0000 7f6392f8b700 1 -- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6370098780 msgr2=0x7f637009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:53.948 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.947+0000 7f6392f8b700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6370098780 0x7f637009ac30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f638c13f1f0 tx=0x7f6378009040 comp rx=0 tx=0).stop 2023-09-30T00:17:53.948 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.947+0000 7f6392f8b700 1 -- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c067e10 msgr2=0x7f638c0a5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:53.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.947+0000 7f6392f8b700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c067e10 0x7f638c0a5fe0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6380006fd0 tx=0x7f6380017000 comp rx=0 tx=0).stop 2023-09-30T00:17:53.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.948+0000 7f6392f8b700 1 -- 172.21.15.94:0/4114896268 shutdown_connections 2023-09-30T00:17:53.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.948+0000 7f6392f8b700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6370098780 0x7f637009ac30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:53.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.948+0000 7f6392f8b700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f638c067e10 0x7f638c0a5fe0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:53.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.948+0000 7f6392f8b700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f638c0a8bb0 0x7f638c1385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:53.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.948+0000 7f6392f8b700 1 --2- 172.21.15.94:0/4114896268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f638c0a06c0 0x7f638c13a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:53.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.948+0000 7f6392f8b700 1 -- 172.21.15.94:0/4114896268 >> 172.21.15.94:0/4114896268 conn(0x7f638c06a270 msgr2=0x7f638c0630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:53.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.948+0000 7f6392f8b700 1 -- 172.21.15.94:0/4114896268 shutdown_connections 2023-09-30T00:17:53.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:53.948+0000 7f6392f8b700 1 -- 172.21.15.94:0/4114896268 wait complete. 2023-09-30T00:17:53.962 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-09-30T00:17:53.962 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-09-30T00:17:54.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.081+0000 7f28f8574700 1 Processor -- start 2023-09-30T00:17:54.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.081+0000 7f28f8574700 1 -- start start 2023-09-30T00:17:54.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.082+0000 7f28f8574700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f28f00b4be0 0x7f28f0143d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:54.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.082+0000 7f28f8574700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f00b4fb0 0x7f28f00a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:54.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.082+0000 7f28f8574700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f0144ba0 con 0x7f28f0144720 2023-09-30T00:17:54.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.082+0000 7f28f8574700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f0144290 con 0x7f28f00b4be0 2023-09-30T00:17:54.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.082+0000 7f28f8574700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00a6fe0 con 0x7f28f00b4fb0 2023-09-30T00:17:54.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.082+0000 7f28f5b0f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f00b4fb0 0x7f28f00a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:54.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.082+0000 7f28f6310700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f28f00b4be0 0x7f28f0143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:54.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.082+0000 7f28f6310700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f28f00b4be0 0x7f28f0143d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33530/0 (socket says 172.21.15.94:33530) 2023-09-30T00:17:54.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.082+0000 7f28f5b0f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f00b4fb0 0x7f28f00a60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51672/0 (socket says 172.21.15.94:51672) 2023-09-30T00:17:54.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.082+0000 7f28f6310700 1 -- 172.21.15.94:0/3900730522 learned_addr learned my addr 172.21.15.94:0/3900730522 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:54.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.082+0000 7f28f6b11700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f28f0144720 0x7f28f00a72a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:32948/0 (socket says 172.21.15.94:32948) 2023-09-30T00:17:54.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.083+0000 7f28f5b0f700 1 -- 172.21.15.94:0/3900730522 >> v2:172.21.15.94:3300/0 conn(0x7f28f00b4be0 msgr2=0x7f28f0143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:54.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.083+0000 7f28f5b0f700 1 --2- 172.21.15.94:0/3900730522 >> v2:172.21.15.94:3300/0 conn(0x7f28f00b4be0 0x7f28f0143d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:54.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.083+0000 7f28f5b0f700 1 -- 172.21.15.94:0/3900730522 >> v1:172.21.15.94:6789/0 conn(0x7f28f0144720 legacy=0x7f28f00a72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:54.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.083+0000 7f28f5b0f700 1 -- 172.21.15.94:0/3900730522 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f00a79b0 con 0x7f28f00b4fb0 2023-09-30T00:17:54.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.083+0000 7f28f530e700 1 -- 172.21.15.94:0/3900730522 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971102359 0 0) 0x7f28e8001110 con 0x7f28f0144720 2023-09-30T00:17:54.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.083+0000 7f28f6310700 1 --2- 172.21.15.94:0/3900730522 >> v2:172.21.15.94:3300/0 conn(0x7f28f00b4be0 0x7f28f0143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:54.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.083+0000 7f28f5b0f700 1 --2- 172.21.15.94:0/3900730522 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f00b4fb0 0x7f28f00a60b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f28e00098e0 tx=0x7f28e00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b685d039bc3ea72e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:54.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.083+0000 7f28f530e700 1 -- 172.21.15.94:0/3900730522 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e000b640 con 0x7f28f00b4fb0 2023-09-30T00:17:54.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.083+0000 7f28f530e700 1 -- 172.21.15.94:0/3900730522 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e000b7a0 con 0x7f28f00b4fb0 2023-09-30T00:17:54.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.083+0000 7f28f530e700 1 -- 172.21.15.94:0/3900730522 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e000bb40 con 0x7f28f00b4fb0 2023-09-30T00:17:54.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.083+0000 7f28f8574700 1 -- 172.21.15.94:0/3900730522 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f00b4fb0 msgr2=0x7f28f00a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:54.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.083+0000 7f28f8574700 1 --2- 172.21.15.94:0/3900730522 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f00b4fb0 0x7f28f00a60b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f28e00098e0 tx=0x7f28e00091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:54.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.084+0000 7f28f8574700 1 -- 172.21.15.94:0/3900730522 shutdown_connections 2023-09-30T00:17:54.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.084+0000 7f28f8574700 1 --2- 172.21.15.94:0/3900730522 >> v2:172.21.15.94:3300/0 conn(0x7f28f00b4be0 0x7f28f0143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:54.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.084+0000 7f28f8574700 1 --2- 172.21.15.94:0/3900730522 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f00b4fb0 0x7f28f00a60b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:54.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.084+0000 7f28f8574700 1 -- 172.21.15.94:0/3900730522 >> 172.21.15.94:0/3900730522 conn(0x7f28f006a940 msgr2=0x7f28f006ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:54.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.084+0000 7f28f8574700 1 -- 172.21.15.94:0/3900730522 shutdown_connections 2023-09-30T00:17:54.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.084+0000 7f28f8574700 1 -- 172.21.15.94:0/3900730522 wait complete. 2023-09-30T00:17:54.089 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.084+0000 7f28f8574700 1 Processor -- start 2023-09-30T00:17:54.089 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.084+0000 7f28f8574700 1 -- start start 2023-09-30T00:17:54.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f8574700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28f00b4be0 0x7f28f009b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:54.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f8574700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28f00b4fb0 0x7f28f0153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:54.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f8574700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f0144720 0x7f28f0154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:54.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f8574700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f0154640 con 0x7f28f00b4fb0 2023-09-30T00:17:54.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f8574700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f009b040 con 0x7f28f00b4be0 2023-09-30T00:17:54.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f8574700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f0153770 con 0x7f28f0144720 2023-09-30T00:17:54.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f6b11700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f0144720 0x7f28f0154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:54.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f6b11700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f0144720 0x7f28f0154100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51676/0 (socket says 172.21.15.94:51676) 2023-09-30T00:17:54.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f6b11700 1 -- 172.21.15.94:0/3373681725 learned_addr learned my addr 172.21.15.94:0/3373681725 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:54.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f5b0f700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28f00b4fb0 0x7f28f0153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:54.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f6310700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28f00b4be0 0x7f28f009b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:54.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f6b11700 1 -- 172.21.15.94:0/3373681725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28f00b4be0 msgr2=0x7f28f009b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:54.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f6b11700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28f00b4be0 0x7f28f009b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:54.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f6b11700 1 -- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28f00b4fb0 msgr2=0x7f28f0153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:54.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f6b11700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28f00b4fb0 0x7f28f0153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:54.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.085+0000 7f28f6b11700 1 -- 172.21.15.94:0/3373681725 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec009ae0 con 0x7f28f0144720 2023-09-30T00:17:54.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.086+0000 7f28f6b11700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f0144720 0x7f28f0154100 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f28e800c530 tx=0x7f28e8007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:54.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.086+0000 7f28e67fc700 1 -- 172.21.15.94:0/3373681725 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8005a70 con 0x7f28f0144720 2023-09-30T00:17:54.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.086+0000 7f28e67fc700 1 -- 172.21.15.94:0/3373681725 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e8005bd0 con 0x7f28f0144720 2023-09-30T00:17:54.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.086+0000 7f28f8574700 1 -- 172.21.15.94:0/3373681725 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e0009420 con 0x7f28f0144720 2023-09-30T00:17:54.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.086+0000 7f28e67fc700 1 -- 172.21.15.94:0/3373681725 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e80070d0 con 0x7f28f0144720 2023-09-30T00:17:54.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.086+0000 7f28f8574700 1 -- 172.21.15.94:0/3373681725 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f009bef0 con 0x7f28f0144720 2023-09-30T00:17:54.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.086+0000 7f28f8574700 1 -- 172.21.15.94:0/3373681725 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f0153900 con 0x7f28f0144720 2023-09-30T00:17:54.092 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.088+0000 7f28e67fc700 1 -- 172.21.15.94:0/3373681725 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f28e8007480 con 0x7f28f0144720 2023-09-30T00:17:54.092 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.088+0000 7f28e67fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:54.092 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.088+0000 7f28e67fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:54.092 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.088+0000 7f28e67fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:54.092 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.088+0000 7f28e67fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:54.092 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.088+0000 7f28e67fc700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f28d4098780 0x7f28d409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:54.092 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.088+0000 7f28f6310700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f28d4098780 0x7f28d409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:54.092 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.088+0000 7f28e67fc700 1 -- 172.21.15.94:0/3373681725 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f28e80bfa00 con 0x7f28f0144720 2023-09-30T00:17:54.092 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.089+0000 7f28f6310700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f28d4098780 0x7f28d409ac30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009ab0 tx=0x7f28ec00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:54.093 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.091+0000 7f28e67fc700 1 -- 172.21.15.94:0/3373681725 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f28e808d0b0 con 0x7f28f0144720 2023-09-30T00:17:54.318 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.317+0000 7f28f8574700 1 -- 172.21.15.94:0/3373681725 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f28f0144ba0 con 0x7f28f0144720 2023-09-30T00:17:54.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.949+0000 7f28e67fc700 1 -- 172.21.15.94:0/3373681725 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f28f0144ba0 con 0x7f28f0144720 2023-09-30T00:17:54.952 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 10 mgrc shutdown 2023-09-30T00:17:54.952 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 1 -- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f28d4098780 msgr2=0x7f28d409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:54.952 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f28d4098780 0x7f28d409ac30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009ab0 tx=0x7f28ec00b040 comp rx=0 tx=0).stop 2023-09-30T00:17:54.952 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 1 -- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f0144720 msgr2=0x7f28f0154100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:54.952 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f0144720 0x7f28f0154100 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f28e800c530 tx=0x7f28e8007f90 comp rx=0 tx=0).stop 2023-09-30T00:17:54.952 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 1 -- 172.21.15.94:0/3373681725 shutdown_connections 2023-09-30T00:17:54.953 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f28d4098780 0x7f28d409ac30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:54.953 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f28f0144720 0x7f28f0154100 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:54.953 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28f00b4be0 0x7f28f009b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:54.953 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 1 --2- 172.21.15.94:0/3373681725 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28f00b4fb0 0x7f28f0153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:54.953 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 1 -- 172.21.15.94:0/3373681725 >> 172.21.15.94:0/3373681725 conn(0x7f28f006a940 msgr2=0x7f28f0067d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:54.953 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 1 -- 172.21.15.94:0/3373681725 shutdown_connections 2023-09-30T00:17:54.953 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:54.951+0000 7f28f8574700 1 -- 172.21.15.94:0/3373681725 wait complete. 2023-09-30T00:17:54.964 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-09-30T00:17:54.964 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-30T00:17:55.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.077+0000 7fac06215700 1 Processor -- start 2023-09-30T00:17:55.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.078+0000 7fac06215700 1 -- start start 2023-09-30T00:17:55.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.078+0000 7fac06215700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a6540 0x7fac000b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.078+0000 7fac06215700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fac000a5770 0x7fac000a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.078+0000 7fac06215700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac000a6910 con 0x7fac00141b40 2023-09-30T00:17:55.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.078+0000 7fac06215700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac001427d0 con 0x7fac000a5770 2023-09-30T00:17:55.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.078+0000 7fac06215700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac000b55b0 con 0x7fac000a6540 2023-09-30T00:17:55.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.078+0000 7fac05213700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a6540 0x7fac000b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.078+0000 7fac04a12700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fac000a5770 0x7fac000a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.078+0000 7fac05213700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a6540 0x7fac000b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:38984/0 (socket says 172.21.15.94:38984) 2023-09-30T00:17:55.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.078+0000 7fac05213700 1 -- 172.21.15.94:0/2783109528 learned_addr learned my addr 172.21.15.94:0/2783109528 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:55.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.078+0000 7fac04a12700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fac000a5770 0x7fac000a5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50458/0 (socket says 172.21.15.94:50458) 2023-09-30T00:17:55.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fac05213700 1 -- 172.21.15.94:0/2783109528 >> v2:172.21.15.94:3300/0 conn(0x7fac000a5770 msgr2=0x7fac000a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:55.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fac05213700 1 --2- 172.21.15.94:0/2783109528 >> v2:172.21.15.94:3300/0 conn(0x7fac000a5770 0x7fac000a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fac05213700 1 -- 172.21.15.94:0/2783109528 >> v1:172.21.15.121:6789/0 conn(0x7fac00141b40 legacy=0x7fac00141fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:55.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fac05213700 1 -- 172.21.15.94:0/2783109528 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac000a51f0 con 0x7fac000a6540 2023-09-30T00:17:55.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fac05213700 1 --2- 172.21.15.94:0/2783109528 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a6540 0x7fac000b4f20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fabfc009f20 tx=0x7fabfc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd3790e1e136fabb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:55.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fabf7fff700 1 -- 172.21.15.94:0/2783109528 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc00e070 con 0x7fac000a6540 2023-09-30T00:17:55.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fabf7fff700 1 -- 172.21.15.94:0/2783109528 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc00ba20 con 0x7fac000a6540 2023-09-30T00:17:55.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fabf7fff700 1 -- 172.21.15.94:0/2783109528 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0092e0 con 0x7fac000a6540 2023-09-30T00:17:55.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fac06215700 1 -- 172.21.15.94:0/2783109528 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a6540 msgr2=0x7fac000b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fac06215700 1 --2- 172.21.15.94:0/2783109528 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a6540 0x7fac000b4f20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fabfc009f20 tx=0x7fabfc009840 comp rx=0 tx=0).stop 2023-09-30T00:17:55.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fac06215700 1 -- 172.21.15.94:0/2783109528 shutdown_connections 2023-09-30T00:17:55.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fac06215700 1 --2- 172.21.15.94:0/2783109528 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a6540 0x7fac000b4f20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fac06215700 1 --2- 172.21.15.94:0/2783109528 >> v2:172.21.15.94:3300/0 conn(0x7fac000a5770 0x7fac000a5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.079+0000 7fac06215700 1 -- 172.21.15.94:0/2783109528 >> 172.21.15.94:0/2783109528 conn(0x7fac0006c8a0 msgr2=0x7fac0006cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:55.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.080+0000 7fac06215700 1 -- 172.21.15.94:0/2783109528 shutdown_connections 2023-09-30T00:17:55.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.080+0000 7fac06215700 1 -- 172.21.15.94:0/2783109528 wait complete. 2023-09-30T00:17:55.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.080+0000 7fac06215700 1 Processor -- start 2023-09-30T00:17:55.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.080+0000 7fac06215700 1 -- start start 2023-09-30T00:17:55.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.081+0000 7fac06215700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a5770 0x7fac000b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.081+0000 7fac06215700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac000a6540 0x7fac000b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.081+0000 7fac06215700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac00141b40 0x7fac000b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.081+0000 7fac06215700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac000b1600 con 0x7fac00141b40 2023-09-30T00:17:55.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.081+0000 7fac06215700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0014e0c0 con 0x7fac000a6540 2023-09-30T00:17:55.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.081+0000 7fac06215700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac001504b0 con 0x7fac000a5770 2023-09-30T00:17:55.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.081+0000 7fac05213700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a5770 0x7fac000b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.081+0000 7fac05213700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a5770 0x7fac000b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:38988/0 (socket says 172.21.15.94:38988) 2023-09-30T00:17:55.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.081+0000 7fac05a14700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac00141b40 0x7fac000b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.081+0000 7fac05213700 1 -- 172.21.15.94:0/589327534 learned_addr learned my addr 172.21.15.94:0/589327534 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.081+0000 7fac04a12700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac000a6540 0x7fac000b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.082+0000 7fac05213700 1 -- 172.21.15.94:0/589327534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac000a6540 msgr2=0x7fac000b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.082+0000 7fac05213700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac000a6540 0x7fac000b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.082+0000 7fac05213700 1 -- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac00141b40 msgr2=0x7fac000b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.082+0000 7fac05213700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac00141b40 0x7fac000b0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.082+0000 7fac05213700 1 -- 172.21.15.94:0/589327534 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc009ae0 con 0x7fac000a5770 2023-09-30T00:17:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.082+0000 7fac05a14700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac00141b40 0x7fac000b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.082+0000 7fac05213700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a5770 0x7fac000b32a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fac000b56f0 tx=0x7fabfc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.082+0000 7fabf57fa700 1 -- 172.21.15.94:0/589327534 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc012850 con 0x7fac000a5770 2023-09-30T00:17:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.082+0000 7fabf57fa700 1 -- 172.21.15.94:0/589327534 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc0129d0 con 0x7fac000a5770 2023-09-30T00:17:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.082+0000 7fac06215700 1 -- 172.21.15.94:0/589327534 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac000b37e0 con 0x7fac000a5770 2023-09-30T00:17:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.082+0000 7fabf57fa700 1 -- 172.21.15.94:0/589327534 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc012d90 con 0x7fac000a5770 2023-09-30T00:17:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.082+0000 7fac06215700 1 -- 172.21.15.94:0/589327534 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac000adf10 con 0x7fac000a5770 2023-09-30T00:17:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.083+0000 7fac06215700 1 -- 172.21.15.94:0/589327534 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0004c380 con 0x7fac000a5770 2023-09-30T00:17:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.084+0000 7fabf57fa700 1 -- 172.21.15.94:0/589327534 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fabfc01d630 con 0x7fac000a5770 2023-09-30T00:17:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.084+0000 7fabf57fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.084+0000 7fabf57fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.084+0000 7fabf57fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.084+0000 7fabf57fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.084+0000 7fabf57fa700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fabe4098780 0x7fabe409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.085+0000 7fac04a12700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fabe4098780 0x7fabe409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.089 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.085+0000 7fabf57fa700 1 -- 172.21.15.94:0/589327534 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fabfc0c1910 con 0x7fac000a5770 2023-09-30T00:17:55.089 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.085+0000 7fac04a12700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fabe4098780 0x7fabe409ac30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fac0006a290 tx=0x7fabf00091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:55.089 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.089+0000 7fabf57fa700 1 -- 172.21.15.94:0/589327534 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fabfc08f050 con 0x7fac000a5770 2023-09-30T00:17:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.312+0000 7fac06215700 1 -- 172.21.15.94:0/589327534 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac0012f8a0 con 0x7fac000a5770 2023-09-30T00:17:55.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.312+0000 7fabf57fa700 1 -- 172.21.15.94:0/589327534 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+3835 (secure 0 0 0) 0x7fabfc092670 con 0x7fac000a5770 2023-09-30T00:17:55.313 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:55.313 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4187,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6813/1048306399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":1048306399},{"type":"v1","addr":"172.21.15.94:6813","nonce":1048306399}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4231,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6825/549674001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":549674001},{"type":"v1","addr":"172.21.15.121:6825","nonce":549674001}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-09-30T00:17:51.937528+0000","modified":"2023-09-30T00:17:54.944027+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4239":{"gid":4239,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.94:6811/2461396790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2461396790},{"type":"v1","addr":"172.21.15.94:6811","nonce":2461396790}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-09-30T00:17:55.315 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.314+0000 7fac06215700 10 mgrc shutdown 2023-09-30T00:17:55.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.314+0000 7fac06215700 1 -- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fabe4098780 msgr2=0x7fabe409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.314+0000 7fac06215700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fabe4098780 0x7fabe409ac30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fac0006a290 tx=0x7fabf00091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.314+0000 7fac06215700 1 -- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a5770 msgr2=0x7fac000b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.314+0000 7fac06215700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a5770 0x7fac000b32a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fac000b56f0 tx=0x7fabfc017000 comp rx=0 tx=0).stop 2023-09-30T00:17:55.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.314+0000 7fac06215700 1 -- 172.21.15.94:0/589327534 shutdown_connections 2023-09-30T00:17:55.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.314+0000 7fac06215700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fabe4098780 0x7fabe409ac30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.314+0000 7fac06215700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fac000a5770 0x7fac000b32a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.314+0000 7fac06215700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac00141b40 0x7fac000b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.314+0000 7fac06215700 1 --2- 172.21.15.94:0/589327534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac000a6540 0x7fac000b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.315+0000 7fac06215700 1 -- 172.21.15.94:0/589327534 >> 172.21.15.94:0/589327534 conn(0x7fac0006c8a0 msgr2=0x7fac0012fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:55.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.315+0000 7fac06215700 1 -- 172.21.15.94:0/589327534 shutdown_connections 2023-09-30T00:17:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.315+0000 7fac06215700 1 -- 172.21.15.94:0/589327534 wait complete. 2023-09-30T00:17:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 8 2023-09-30T00:17:55.327 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:17:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.439+0000 7fddbf83a700 1 Processor -- start 2023-09-30T00:17:55.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.439+0000 7fddbf83a700 1 -- start start 2023-09-30T00:17:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.440+0000 7fddbf83a700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fddb806a460 0x7fddb80b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.440+0000 7fddbf83a700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb8069760 con 0x7fddb80a46c0 2023-09-30T00:17:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.440+0000 7fddbf83a700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb80a4a90 con 0x7fddb80b5680 2023-09-30T00:17:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.440+0000 7fddbf83a700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb80a57f0 con 0x7fddb806a460 2023-09-30T00:17:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.440+0000 7fddbd5d6700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fddb80a46c0 0x7fddb8068c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50192/0 (socket says 172.21.15.94:50192) 2023-09-30T00:17:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.440+0000 7fddbd5d6700 1 -- 172.21.15.94:0/2605491367 learned_addr learned my addr 172.21.15.94:0/2605491367 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:17:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.440+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831650329 0 0) 0x7fddb8069760 con 0x7fddb80a46c0 2023-09-30T00:17:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.440+0000 7fddbcdd5700 1 --2- 172.21.15.94:0/2605491367 >> v2:172.21.15.121:3300/0 conn(0x7fddb806a460 0x7fddb80b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.440+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda4003570 con 0x7fddb80a46c0 2023-09-30T00:17:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1871243512 0 0) 0x7fdda4003570 con 0x7fddb80a46c0 2023-09-30T00:17:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 >> v2:172.21.15.121:3300/0 conn(0x7fddb806a460 msgr2=0x7fddb80b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:55.447 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 --2- 172.21.15.94:0/2605491367 >> v2:172.21.15.121:3300/0 conn(0x7fddb806a460 0x7fddb80b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 >> v1:172.21.15.121:6789/0 conn(0x7fddb80b5680 legacy=0x7fddb814b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb8069c80 con 0x7fddb80a46c0 2023-09-30T00:17:55.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3153706020 0 0) 0x7fddb40017f0 con 0x7fddb80a46c0 2023-09-30T00:17:55.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 >> v1:172.21.15.94:6789/0 conn(0x7fddb80a46c0 legacy=0x7fddb8068c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 --2- 172.21.15.94:0/2605491367 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdda4003a90 0x7fdda4003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 --2- 172.21.15.94:0/2605491367 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdda4004810 0x7fdda4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 --2- 172.21.15.94:0/2605491367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdda4005210 0x7fdda400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb80a57f0 con 0x7fdda4005210 2023-09-30T00:17:55.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda400c130 con 0x7fdda4003a90 2023-09-30T00:17:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda400c2c0 con 0x7fdda4004810 2023-09-30T00:17:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddbcdd5700 1 --2- 172.21.15.94:0/2605491367 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdda4004810 0x7fdda4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddbddd7700 1 --2- 172.21.15.94:0/2605491367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdda4005210 0x7fdda400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddbd5d6700 1 --2- 172.21.15.94:0/2605491367 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdda4003a90 0x7fdda4003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddbcdd5700 1 -- 172.21.15.94:0/2605491367 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdda4003a90 msgr2=0x7fdda4003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddbcdd5700 1 --2- 172.21.15.94:0/2605491367 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdda4003a90 0x7fdda4003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddbcdd5700 1 -- 172.21.15.94:0/2605491367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdda4005210 msgr2=0x7fdda400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddbcdd5700 1 --2- 172.21.15.94:0/2605491367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdda4005210 0x7fdda400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddbcdd5700 1 -- 172.21.15.94:0/2605491367 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda4003570 con 0x7fdda4004810 2023-09-30T00:17:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddbddd7700 1 --2- 172.21.15.94:0/2605491367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdda4005210 0x7fdda400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:55.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddbddd7700 1 -- 172.21.15.94:0/2605491367 reap_dead start 2023-09-30T00:17:55.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddbcdd5700 1 --2- 172.21.15.94:0/2605491367 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdda4004810 0x7fdda4004c80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdda8009c30 tx=0x7fdda80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b2f84bbacaafd2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:55.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda80095b0 con 0x7fdda4004810 2023-09-30T00:17:55.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.441+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda800e710 con 0x7fdda4004810 2023-09-30T00:17:55.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.442+0000 7fddaffff700 1 -- 172.21.15.94:0/2605491367 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda800ead0 con 0x7fdda4004810 2023-09-30T00:17:55.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.442+0000 7fddbf83a700 1 -- 172.21.15.94:0/2605491367 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdda4004810 msgr2=0x7fdda4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.442+0000 7fddbf83a700 1 --2- 172.21.15.94:0/2605491367 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdda4004810 0x7fdda4004c80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdda8009c30 tx=0x7fdda80091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.442+0000 7fddbf83a700 1 -- 172.21.15.94:0/2605491367 shutdown_connections 2023-09-30T00:17:55.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.442+0000 7fddbf83a700 1 --2- 172.21.15.94:0/2605491367 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdda4004810 0x7fdda4004c80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.451 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.442+0000 7fddbf83a700 1 -- 172.21.15.94:0/2605491367 >> 172.21.15.94:0/2605491367 conn(0x7fddb806cbf0 msgr2=0x7fddb806cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:55.451 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.442+0000 7fddbf83a700 1 -- 172.21.15.94:0/2605491367 shutdown_connections 2023-09-30T00:17:55.451 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.442+0000 7fddbf83a700 1 -- 172.21.15.94:0/2605491367 wait complete. 2023-09-30T00:17:55.451 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.443+0000 7fddbf83a700 1 Processor -- start 2023-09-30T00:17:55.451 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.443+0000 7fddbf83a700 1 -- start start 2023-09-30T00:17:55.451 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbf83a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdda4004810 0x7fddb8140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.451 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbf83a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddb80b03a0 0x7fddb80b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.451 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbf83a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fddb811d840 0x7fddb811dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.451 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbf83a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb8146730 con 0x7fdda4004810 2023-09-30T00:17:55.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbf83a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb811c8b0 con 0x7fddb80b03a0 2023-09-30T00:17:55.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbf83a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb811e1f0 con 0x7fddb811d840 2023-09-30T00:17:55.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbd5d6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdda4004810 0x7fddb8140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbddd7700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fddb811d840 0x7fddb811dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbd5d6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdda4004810 0x7fddb8140ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50484/0 (socket says 172.21.15.94:50484) 2023-09-30T00:17:55.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbddd7700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fddb811d840 0x7fddb811dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:38994/0 (socket says 172.21.15.94:38994) 2023-09-30T00:17:55.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbd5d6700 1 -- 172.21.15.94:0/513474085 learned_addr learned my addr 172.21.15.94:0/513474085 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:55.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbcdd5700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddb80b03a0 0x7fddb80b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbddd7700 1 -- 172.21.15.94:0/513474085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddb80b03a0 msgr2=0x7fddb80b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbddd7700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddb80b03a0 0x7fddb80b11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbddd7700 1 -- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdda4004810 msgr2=0x7fddb8140ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbddd7700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdda4004810 0x7fddb8140ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbddd7700 1 -- 172.21.15.94:0/513474085 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb80b0830 con 0x7fddb811d840 2023-09-30T00:17:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbcdd5700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddb80b03a0 0x7fddb80b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:17:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbd5d6700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdda4004810 0x7fddb8140ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddbddd7700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fddb811d840 0x7fddb811dcb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fddb80b4c00 tx=0x7fddb0009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.444+0000 7fddad7fa700 1 -- 172.21.15.94:0/513474085 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0011660 con 0x7fddb811d840 2023-09-30T00:17:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.445+0000 7fddad7fa700 1 -- 172.21.15.94:0/513474085 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb00117c0 con 0x7fddb811d840 2023-09-30T00:17:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.445+0000 7fddbf83a700 1 -- 172.21.15.94:0/513474085 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb80b0ad0 con 0x7fddb811d840 2023-09-30T00:17:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.445+0000 7fddad7fa700 1 -- 172.21.15.94:0/513474085 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0011b80 con 0x7fddb811d840 2023-09-30T00:17:55.454 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.445+0000 7fddbf83a700 1 -- 172.21.15.94:0/513474085 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb81415f0 con 0x7fddb811d840 2023-09-30T00:17:55.454 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.445+0000 7fddbf83a700 1 -- 172.21.15.94:0/513474085 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb8141780 con 0x7fddb811d840 2023-09-30T00:17:55.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.446+0000 7fddad7fa700 1 -- 172.21.15.94:0/513474085 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fddb001d460 con 0x7fddb811d840 2023-09-30T00:17:55.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.446+0000 7fddad7fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:55.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.447+0000 7fddad7fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:55.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.447+0000 7fddad7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:55.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.447+0000 7fddad7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:55.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.447+0000 7fddad7fa700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdd9c098780 0x7fdd9c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.447+0000 7fddbd5d6700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdd9c098780 0x7fdd9c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.447+0000 7fddad7fa700 1 -- 172.21.15.94:0/513474085 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fddb00c0b60 con 0x7fddb811d840 2023-09-30T00:17:55.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.447+0000 7fddbd5d6700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdd9c098780 0x7fdd9c09ac30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fddb4004fd0 tx=0x7fddb4006040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:55.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.450+0000 7fddad7fa700 1 -- 172.21.15.94:0/513474085 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fddb008e2a0 con 0x7fddb811d840 2023-09-30T00:17:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.632+0000 7fddbf83a700 1 -- 172.21.15.94:0/513474085 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddb806fbd0 con 0x7fddb811d840 2023-09-30T00:17:55.633 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.632+0000 7fddad7fa700 1 -- 172.21.15.94:0/513474085 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12197 (secure 0 0 0) 0x7fddb00918c0 con 0x7fddb811d840 2023-09-30T00:17:55.633 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:55.633 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":19,"fsid":"3c891173-587e-4c6f-9c6a-04ec750eedad","created":"2023-09-30T00:17:34.347592+0000","modified":"2023-09-30T00:17:52.938964+0000","last_up_change":"2023-09-30T00:17:42.879328+0000","last_in_change":"2023-09-30T00:17:35.583427+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T00:17:43.481613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T00:17:45.892643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-09-30T00:17:50.220028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-09-30T00:17:50.911066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c508b402-f8ce-4461-823b-9e35fcd8d40f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":84364},{"type":"v1","addr":"172.21.15.94:6801","nonce":84364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":84364},{"type":"v1","addr":"172.21.15.94:6803","nonce":84364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":84364},{"type":"v1","addr":"172.21.15.94:6807","nonce":84364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":84364},{"type":"v1","addr":"172.21.15.94:6805","nonce":84364}]},"public_addr":"172.21.15.94:6801/84364","cluster_addr":"172.21.15.94:6803/84364","heartbeat_back_addr":"172.21.15.94:6807/84364","heartbeat_front_addr":"172.21.15.94:6805/84364","state":["exists","up"]},{"osd":1,"uuid":"e96ecdf0-bb38-4eb1-be99-54f813c12888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":86850},{"type":"v1","addr":"172.21.15.121:6817","nonce":86850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":86850},{"type":"v1","addr":"172.21.15.121:6819","nonce":86850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":86850},{"type":"v1","addr":"172.21.15.121:6823","nonce":86850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":86850},{"type":"v1","addr":"172.21.15.121:6821","nonce":86850}]},"public_addr":"172.21.15.121:6817/86850","cluster_addr":"172.21.15.121:6819/86850","heartbeat_back_addr":"172.21.15.121:6823/86850","heartbeat_front_addr":"172.21.15.121:6821/86850","state":["exists","up"]},{"osd":2,"uuid":"68700ae4-764c-4dff-9cf2-5f33132cd0d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":86820},{"type":"v1","addr":"172.21.15.121:6801","nonce":86820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":86820},{"type":"v1","addr":"172.21.15.121:6803","nonce":86820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":86820},{"type":"v1","addr":"172.21.15.121:6810","nonce":86820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":86820},{"type":"v1","addr":"172.21.15.121:6806","nonce":86820}]},"public_addr":"172.21.15.121:6801/86820","cluster_addr":"172.21.15.121:6803/86820","heartbeat_back_addr":"172.21.15.121:6810/86820","heartbeat_front_addr":"172.21.15.121:6806/86820","state":["exists","up"]},{"osd":3,"uuid":"54f17ec8-2561-42e8-bbd7-262130ffdac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":86821},{"type":"v1","addr":"172.21.15.121:6807","nonce":86821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":86821},{"type":"v1","addr":"172.21.15.121:6811","nonce":86821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":86821},{"type":"v1","addr":"172.21.15.121:6815","nonce":86821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":86821},{"type":"v1","addr":"172.21.15.121:6813","nonce":86821}]},"public_addr":"172.21.15.121:6807/86821","cluster_addr":"172.21.15.121:6811/86821","heartbeat_back_addr":"172.21.15.121:6815/86821","heartbeat_front_addr":"172.21.15.121:6813/86821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:39.951865+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.769149+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.738994+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.748197+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:17:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.634+0000 7fddbf83a700 10 mgrc shutdown 2023-09-30T00:17:55.635 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.635+0000 7fddbf83a700 1 -- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdd9c098780 msgr2=0x7fdd9c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.635+0000 7fddbf83a700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdd9c098780 0x7fdd9c09ac30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fddb4004fd0 tx=0x7fddb4006040 comp rx=0 tx=0).stop 2023-09-30T00:17:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.635+0000 7fddbf83a700 1 -- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fddb811d840 msgr2=0x7fddb811dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.635+0000 7fddbf83a700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fddb811d840 0x7fddb811dcb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fddb80b4c00 tx=0x7fddb0009040 comp rx=0 tx=0).stop 2023-09-30T00:17:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.635+0000 7fddbf83a700 1 -- 172.21.15.94:0/513474085 shutdown_connections 2023-09-30T00:17:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.635+0000 7fddbf83a700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdd9c098780 0x7fdd9c09ac30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.635+0000 7fddbf83a700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fddb811d840 0x7fddb811dcb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.635+0000 7fddbf83a700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdda4004810 0x7fddb8140ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.635+0000 7fddbf83a700 1 --2- 172.21.15.94:0/513474085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddb80b03a0 0x7fddb80b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.635+0000 7fddbf83a700 1 -- 172.21.15.94:0/513474085 >> 172.21.15.94:0/513474085 conn(0x7fddb806cbf0 msgr2=0x7fddb80630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.635+0000 7fddbf83a700 1 -- 172.21.15.94:0/513474085 shutdown_connections 2023-09-30T00:17:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.635+0000 7fddbf83a700 1 -- 172.21.15.94:0/513474085 wait complete. 2023-09-30T00:17:55.647 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-09-30T00:17:55.647 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:17:55.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.760+0000 7f10a74bd700 1 Processor -- start 2023-09-30T00:17:55.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.760+0000 7f10a74bd700 1 -- start start 2023-09-30T00:17:55.763 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a74bd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a00a3ff0 0x7f10a00b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.763 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a74bd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f10a009ca70 0x7f10a009ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.763 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a74bd700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a006ae70 con 0x7f10a006aaa0 2023-09-30T00:17:55.763 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a74bd700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a00b48c0 con 0x7f10a009ca70 2023-09-30T00:17:55.763 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a74bd700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00a45c0 con 0x7f10a00a3ff0 2023-09-30T00:17:55.763 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a4a58700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a00a3ff0 0x7f10a00b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.763 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a5a5a700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f10a009ca70 0x7f10a009ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.763 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a5a5a700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f10a009ca70 0x7f10a009ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50500/0 (socket says 172.21.15.94:50500) 2023-09-30T00:17:55.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a4a58700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a00a3ff0 0x7f10a00b24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39006/0 (socket says 172.21.15.94:39006) 2023-09-30T00:17:55.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a5a5a700 1 -- 172.21.15.94:0/902018402 learned_addr learned my addr 172.21.15.94:0/902018402 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:55.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a4a58700 1 -- 172.21.15.94:0/902018402 >> v2:172.21.15.94:3300/0 conn(0x7f10a009ca70 msgr2=0x7f10a009ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:55.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a4a58700 1 --2- 172.21.15.94:0/902018402 >> v2:172.21.15.94:3300/0 conn(0x7f10a009ca70 0x7f10a009ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a4a58700 1 -- 172.21.15.94:0/902018402 >> v1:172.21.15.121:6789/0 conn(0x7f10a006aaa0 legacy=0x7f10a00b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a4a58700 1 -- 172.21.15.94:0/902018402 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a00a4880 con 0x7f10a00a3ff0 2023-09-30T00:17:55.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.761+0000 7f10a5a5a700 1 --2- 172.21.15.94:0/902018402 >> v2:172.21.15.94:3300/0 conn(0x7f10a009ca70 0x7f10a009ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:55.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.762+0000 7f10a4a58700 1 --2- 172.21.15.94:0/902018402 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a00a3ff0 0x7f10a00b24b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f10900098e0 tx=0x7f10900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55b8cf0b59601a5a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:55.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.762+0000 7f1097fff700 1 -- 172.21.15.94:0/902018402 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109000b640 con 0x7f10a00a3ff0 2023-09-30T00:17:55.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.762+0000 7f1097fff700 1 -- 172.21.15.94:0/902018402 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109000b7a0 con 0x7f10a00a3ff0 2023-09-30T00:17:55.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.762+0000 7f1097fff700 1 -- 172.21.15.94:0/902018402 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109000bb40 con 0x7f10a00a3ff0 2023-09-30T00:17:55.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.762+0000 7f10a74bd700 1 -- 172.21.15.94:0/902018402 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a00a3ff0 msgr2=0x7f10a00b24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.765 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.762+0000 7f10a74bd700 1 --2- 172.21.15.94:0/902018402 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a00a3ff0 0x7f10a00b24b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f10900098e0 tx=0x7f10900091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.765 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.762+0000 7f10a74bd700 1 -- 172.21.15.94:0/902018402 shutdown_connections 2023-09-30T00:17:55.765 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.762+0000 7f10a74bd700 1 --2- 172.21.15.94:0/902018402 >> v2:172.21.15.94:3300/0 conn(0x7f10a009ca70 0x7f10a009ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.765 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.762+0000 7f10a74bd700 1 --2- 172.21.15.94:0/902018402 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a00a3ff0 0x7f10a00b24b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.765 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.762+0000 7f10a74bd700 1 -- 172.21.15.94:0/902018402 >> 172.21.15.94:0/902018402 conn(0x7f10a0068730 msgr2=0x7f10a0068b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:55.765 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.762+0000 7f10a74bd700 1 -- 172.21.15.94:0/902018402 shutdown_connections 2023-09-30T00:17:55.765 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.762+0000 7f10a74bd700 1 -- 172.21.15.94:0/902018402 wait complete. 2023-09-30T00:17:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.763+0000 7f10a74bd700 1 Processor -- start 2023-09-30T00:17:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.763+0000 7f10a74bd700 1 -- start start 2023-09-30T00:17:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a74bd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a006aaa0 0x7f10a00a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a74bd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10a009ca70 0x7f10a01400e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a74bd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10a00a3ff0 0x7f10a012c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a74bd700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00a2470 con 0x7f10a00a3ff0 2023-09-30T00:17:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a74bd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00a26b0 con 0x7f10a009ca70 2023-09-30T00:17:55.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a74bd700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00a2970 con 0x7f10a006aaa0 2023-09-30T00:17:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a5259700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a006aaa0 0x7f10a00a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a5a5a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10a00a3ff0 0x7f10a012c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a5259700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a006aaa0 0x7f10a00a1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39018/0 (socket says 172.21.15.94:39018) 2023-09-30T00:17:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a5259700 1 -- 172.21.15.94:0/857079700 learned_addr learned my addr 172.21.15.94:0/857079700 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a4a58700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10a009ca70 0x7f10a01400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a5a5a700 1 -- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a006aaa0 msgr2=0x7f10a00a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a5a5a700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a006aaa0 0x7f10a00a1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a5a5a700 1 -- 172.21.15.94:0/857079700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10a009ca70 msgr2=0x7f10a01400e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a5a5a700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10a009ca70 0x7f10a01400e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a5a5a700 1 -- 172.21.15.94:0/857079700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10980094f0 con 0x7f10a00a3ff0 2023-09-30T00:17:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.764+0000 7f10a5a5a700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10a00a3ff0 0x7f10a012c3d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f109800ea20 tx=0x7f1098009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.765+0000 7f10957fa700 1 -- 172.21.15.94:0/857079700 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109800ee60 con 0x7f10a00a3ff0 2023-09-30T00:17:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.765+0000 7f10957fa700 1 -- 172.21.15.94:0/857079700 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109800c850 con 0x7f10a00a3ff0 2023-09-30T00:17:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.765+0000 7f10a74bd700 1 -- 172.21.15.94:0/857079700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1090009420 con 0x7f10a00a3ff0 2023-09-30T00:17:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.765+0000 7f10957fa700 1 -- 172.21.15.94:0/857079700 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109800cbd0 con 0x7f10a00a3ff0 2023-09-30T00:17:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.765+0000 7f10a74bd700 1 -- 172.21.15.94:0/857079700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a0140620 con 0x7f10a00a3ff0 2023-09-30T00:17:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.765+0000 7f10a74bd700 1 -- 172.21.15.94:0/857079700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a00b23b0 con 0x7f10a00a3ff0 2023-09-30T00:17:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.766+0000 7f10957fa700 1 -- 172.21.15.94:0/857079700 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f1098013470 con 0x7f10a00a3ff0 2023-09-30T00:17:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.767+0000 7f10957fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.767+0000 7f10957fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:55.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.767+0000 7f10957fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.767+0000 7f10957fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.767+0000 7f10957fa700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1084098780 0x7f108409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.767+0000 7f10a5259700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1084098780 0x7f108409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.767+0000 7f10957fa700 1 -- 172.21.15.94:0/857079700 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f10980c0b20 con 0x7f10a00a3ff0 2023-09-30T00:17:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.767+0000 7f10a5259700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1084098780 0x7f108409ac30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f109c007fd0 tx=0x7f109c0017b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.771+0000 7f10957fa700 1 -- 172.21.15.94:0/857079700 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f109808e260 con 0x7f10a00a3ff0 2023-09-30T00:17:55.952 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.952+0000 7f10a74bd700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-30T00:17:55.952 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.952+0000 7f10a74bd700 1 -- 172.21.15.94:0/857079700 --> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10a00a1690 con 0x7f1084098780 2023-09-30T00:17:55.960 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.959+0000 7f10957fa700 1 -- 172.21.15.94:0/857079700 <== mgr.4106 v2:172.21.15.94:6808/84242 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+268834 (secure 0 0 0) 0x7f109c00ea70 con 0x7f1084098780 2023-09-30T00:17:55.960 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.959+0000 7f10957fa700 20 mgrc handle_command_reply tid 0 r 0 2023-09-30T00:17:55.963 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:55.966 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 10 mgrc shutdown 2023-09-30T00:17:55.966 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 1 -- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1084098780 msgr2=0x7f108409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.966 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1084098780 0x7f108409ac30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f109c007fd0 tx=0x7f109c0017b0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.966 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 1 -- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10a00a3ff0 msgr2=0x7f10a012c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:55.966 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10a00a3ff0 0x7f10a012c3d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f109800ea20 tx=0x7f1098009250 comp rx=0 tx=0).stop 2023-09-30T00:17:55.966 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 1 -- 172.21.15.94:0/857079700 shutdown_connections 2023-09-30T00:17:55.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1084098780 0x7f108409ac30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f10a006aaa0 0x7f10a00a1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10a00a3ff0 0x7f10a012c3d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 1 --2- 172.21.15.94:0/857079700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10a009ca70 0x7f10a01400e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:55.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 1 -- 172.21.15.94:0/857079700 >> 172.21.15.94:0/857079700 conn(0x7f10a0068730 msgr2=0x7f10a0068b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:55.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 1 -- 172.21.15.94:0/857079700 shutdown_connections 2023-09-30T00:17:55.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:55.963+0000 7f10a74bd700 1 -- 172.21.15.94:0/857079700 wait complete. 2023-09-30T00:17:55.967 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2023-09-30T00:17:55.975 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-09-30T00:17:55.851278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":202,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":34184,"kb_used_data":1832,"kb_used_omap":0,"kb_used_meta":32320,"kb_avail":377453176,"statfs":{"total":386547056640,"available":386512052224,"internally_reserved":0,"allocated":1875968,"data_stored":1391950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33095680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.286619"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960193+0000","last_change":"2023-09-30T00:17:51.960193+0000","last_active":"2023-09-30T00:17:51.960193+0000","last_peered":"2023-09-30T00:17:51.960193+0000","last_clean":"2023-09-30T00:17:51.960193+0000","last_became_active":"2023-09-30T00:17:51.959888+0000","last_became_peered":"2023-09-30T00:17:51.959888+0000","last_unstale":"2023-09-30T00:17:51.960193+0000","last_undegraded":"2023-09-30T00:17:51.960193+0000","last_fullsized":"2023-09-30T00:17:51.960193+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:40:42.458928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968577+0000","last_change":"2023-09-30T00:17:51.968577+0000","last_active":"2023-09-30T00:17:51.968577+0000","last_peered":"2023-09-30T00:17:51.968577+0000","last_clean":"2023-09-30T00:17:51.968577+0000","last_became_active":"2023-09-30T00:17:51.968275+0000","last_became_peered":"2023-09-30T00:17:51.968275+0000","last_unstale":"2023-09-30T00:17:51.968577+0000","last_undegraded":"2023-09-30T00:17:51.968577+0000","last_fullsized":"2023-09-30T00:17:51.968577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:19:24.974156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977703+0000","last_change":"2023-09-30T00:17:51.974359+0000","last_active":"2023-09-30T00:17:51.977703+0000","last_peered":"2023-09-30T00:17:51.977703+0000","last_clean":"2023-09-30T00:17:51.977703+0000","last_became_active":"2023-09-30T00:17:51.973789+0000","last_became_peered":"2023-09-30T00:17:51.973789+0000","last_unstale":"2023-09-30T00:17:51.977703+0000","last_undegraded":"2023-09-30T00:17:51.977703+0000","last_fullsized":"2023-09-30T00:17:51.977703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:27.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971859+0000","last_change":"2023-09-30T00:17:51.958127+0000","last_active":"2023-09-30T00:17:51.971859+0000","last_peered":"2023-09-30T00:17:51.971859+0000","last_clean":"2023-09-30T00:17:51.971859+0000","last_became_active":"2023-09-30T00:17:51.957850+0000","last_became_peered":"2023-09-30T00:17:51.957850+0000","last_unstale":"2023-09-30T00:17:51.971859+0000","last_undegraded":"2023-09-30T00:17:51.971859+0000","last_fullsized":"2023-09-30T00:17:51.971859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:48:36.950659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973474+0000","last_change":"2023-09-30T00:17:51.966792+0000","last_active":"2023-09-30T00:17:51.973474+0000","last_peered":"2023-09-30T00:17:51.973474+0000","last_clean":"2023-09-30T00:17:51.973474+0000","last_became_active":"2023-09-30T00:17:51.965887+0000","last_became_peered":"2023-09-30T00:17:51.965887+0000","last_unstale":"2023-09-30T00:17:51.973474+0000","last_undegraded":"2023-09-30T00:17:51.973474+0000","last_fullsized":"2023-09-30T00:17:51.973474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973636+0000","last_change":"2023-09-30T00:17:51.967175+0000","last_active":"2023-09-30T00:17:51.973636+0000","last_peered":"2023-09-30T00:17:51.973636+0000","last_clean":"2023-09-30T00:17:51.973636+0000","last_became_active":"2023-09-30T00:17:51.966408+0000","last_became_peered":"2023-09-30T00:17:51.966408+0000","last_unstale":"2023-09-30T00:17:51.973636+0000","last_undegraded":"2023-09-30T00:17:51.973636+0000","last_fullsized":"2023-09-30T00:17:51.973636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:47:56.029102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972783+0000","last_change":"2023-09-30T00:17:51.964264+0000","last_active":"2023-09-30T00:17:51.972783+0000","last_peered":"2023-09-30T00:17:51.972783+0000","last_clean":"2023-09-30T00:17:51.972783+0000","last_became_active":"2023-09-30T00:17:51.962740+0000","last_became_peered":"2023-09-30T00:17:51.962740+0000","last_unstale":"2023-09-30T00:17:51.972783+0000","last_undegraded":"2023-09-30T00:17:51.972783+0000","last_fullsized":"2023-09-30T00:17:51.972783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:52:26.654463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975296+0000","last_change":"2023-09-30T00:17:51.967599+0000","last_active":"2023-09-30T00:17:51.975296+0000","last_peered":"2023-09-30T00:17:51.975296+0000","last_clean":"2023-09-30T00:17:51.975296+0000","last_became_active":"2023-09-30T00:17:51.961762+0000","last_became_peered":"2023-09-30T00:17:51.961762+0000","last_unstale":"2023-09-30T00:17:51.975296+0000","last_undegraded":"2023-09-30T00:17:51.975296+0000","last_fullsized":"2023-09-30T00:17:51.975296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:35:31.620523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570714+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:14:31.363561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972400+0000","last_change":"2023-09-30T00:17:51.963624+0000","last_active":"2023-09-30T00:17:51.972400+0000","last_peered":"2023-09-30T00:17:51.972400+0000","last_clean":"2023-09-30T00:17:51.972400+0000","last_became_active":"2023-09-30T00:17:51.963303+0000","last_became_peered":"2023-09-30T00:17:51.963303+0000","last_unstale":"2023-09-30T00:17:51.972400+0000","last_undegraded":"2023-09-30T00:17:51.972400+0000","last_fullsized":"2023-09-30T00:17:51.972400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971954+0000","last_change":"2023-09-30T00:17:51.961620+0000","last_active":"2023-09-30T00:17:51.971954+0000","last_peered":"2023-09-30T00:17:51.971954+0000","last_clean":"2023-09-30T00:17:51.971954+0000","last_became_active":"2023-09-30T00:17:51.958237+0000","last_became_peered":"2023-09-30T00:17:51.958237+0000","last_unstale":"2023-09-30T00:17:51.971954+0000","last_undegraded":"2023-09-30T00:17:51.971954+0000","last_fullsized":"2023-09-30T00:17:51.971954+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974397+0000","last_change":"2023-09-30T00:17:51.974397+0000","last_active":"2023-09-30T00:17:51.974397+0000","last_peered":"2023-09-30T00:17:51.974397+0000","last_clean":"2023-09-30T00:17:51.974397+0000","last_became_active":"2023-09-30T00:17:51.974025+0000","last_became_peered":"2023-09-30T00:17:51.974025+0000","last_unstale":"2023-09-30T00:17:51.974397+0000","last_undegraded":"2023-09-30T00:17:51.974397+0000","last_fullsized":"2023-09-30T00:17:51.974397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:49:07.723751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570140+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:39:04.507564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976147+0000","last_change":"2023-09-30T00:17:51.970985+0000","last_active":"2023-09-30T00:17:51.976147+0000","last_peered":"2023-09-30T00:17:51.976147+0000","last_clean":"2023-09-30T00:17:51.976147+0000","last_became_active":"2023-09-30T00:17:51.966979+0000","last_became_peered":"2023-09-30T00:17:51.966979+0000","last_unstale":"2023-09-30T00:17:51.976147+0000","last_undegraded":"2023-09-30T00:17:51.976147+0000","last_fullsized":"2023-09-30T00:17:51.976147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:09:28.833739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975911+0000","last_change":"2023-09-30T00:17:51.970398+0000","last_active":"2023-09-30T00:17:51.975911+0000","last_peered":"2023-09-30T00:17:51.975911+0000","last_clean":"2023-09-30T00:17:51.975911+0000","last_became_active":"2023-09-30T00:17:51.961675+0000","last_became_peered":"2023-09-30T00:17:51.961675+0000","last_unstale":"2023-09-30T00:17:51.975911+0000","last_undegraded":"2023-09-30T00:17:51.975911+0000","last_fullsized":"2023-09-30T00:17:51.975911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:33:21.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972718+0000","last_change":"2023-09-30T00:17:51.969081+0000","last_active":"2023-09-30T00:17:51.972718+0000","last_peered":"2023-09-30T00:17:51.972718+0000","last_clean":"2023-09-30T00:17:51.972718+0000","last_became_active":"2023-09-30T00:17:51.966447+0000","last_became_peered":"2023-09-30T00:17:51.966447+0000","last_unstale":"2023-09-30T00:17:51.972718+0000","last_undegraded":"2023-09-30T00:17:51.972718+0000","last_fullsized":"2023-09-30T00:17:51.972718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:19:12.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960350+0000","last_change":"2023-09-30T00:17:51.953980+0000","last_active":"2023-09-30T00:17:51.960350+0000","last_peered":"2023-09-30T00:17:51.960350+0000","last_clean":"2023-09-30T00:17:51.960350+0000","last_became_active":"2023-09-30T00:17:51.952191+0000","last_became_peered":"2023-09-30T00:17:51.952191+0000","last_unstale":"2023-09-30T00:17:51.960350+0000","last_undegraded":"2023-09-30T00:17:51.960350+0000","last_fullsized":"2023-09-30T00:17:51.960350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:52:46.774408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973554+0000","last_change":"2023-09-30T00:17:51.969589+0000","last_active":"2023-09-30T00:17:51.973554+0000","last_peered":"2023-09-30T00:17:51.973554+0000","last_clean":"2023-09-30T00:17:51.973554+0000","last_became_active":"2023-09-30T00:17:51.967300+0000","last_became_peered":"2023-09-30T00:17:51.967300+0000","last_unstale":"2023-09-30T00:17:51.973554+0000","last_undegraded":"2023-09-30T00:17:51.973554+0000","last_fullsized":"2023-09-30T00:17:51.973554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:22:57.623135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568987+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:41:47.152317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972047+0000","last_change":"2023-09-30T00:17:51.961502+0000","last_active":"2023-09-30T00:17:51.972047+0000","last_peered":"2023-09-30T00:17:51.972047+0000","last_clean":"2023-09-30T00:17:51.972047+0000","last_became_active":"2023-09-30T00:17:51.961191+0000","last_became_peered":"2023-09-30T00:17:51.961191+0000","last_unstale":"2023-09-30T00:17:51.972047+0000","last_undegraded":"2023-09-30T00:17:51.972047+0000","last_fullsized":"2023-09-30T00:17:51.972047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:05:37.196886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976121+0000","last_change":"2023-09-30T00:17:51.970789+0000","last_active":"2023-09-30T00:17:51.976121+0000","last_peered":"2023-09-30T00:17:51.976121+0000","last_clean":"2023-09-30T00:17:51.976121+0000","last_became_active":"2023-09-30T00:17:51.967616+0000","last_became_peered":"2023-09-30T00:17:51.967616+0000","last_unstale":"2023-09-30T00:17:51.976121+0000","last_undegraded":"2023-09-30T00:17:51.976121+0000","last_fullsized":"2023-09-30T00:17:51.976121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976127+0000","last_change":"2023-09-30T00:17:51.970797+0000","last_active":"2023-09-30T00:17:51.976127+0000","last_peered":"2023-09-30T00:17:51.976127+0000","last_clean":"2023-09-30T00:17:51.976127+0000","last_became_active":"2023-09-30T00:17:51.963382+0000","last_became_peered":"2023-09-30T00:17:51.963382+0000","last_unstale":"2023-09-30T00:17:51.976127+0000","last_undegraded":"2023-09-30T00:17:51.976127+0000","last_fullsized":"2023-09-30T00:17:51.976127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:35:22.077360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972764+0000","last_change":"2023-09-30T00:17:51.963058+0000","last_active":"2023-09-30T00:17:51.972764+0000","last_peered":"2023-09-30T00:17:51.972764+0000","last_clean":"2023-09-30T00:17:51.972764+0000","last_became_active":"2023-09-30T00:17:51.959450+0000","last_became_peered":"2023-09-30T00:17:51.959450+0000","last_unstale":"2023-09-30T00:17:51.972764+0000","last_undegraded":"2023-09-30T00:17:51.972764+0000","last_fullsized":"2023-09-30T00:17:51.972764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:41:36.585190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975442+0000","last_change":"2023-09-30T00:17:51.967581+0000","last_active":"2023-09-30T00:17:51.975442+0000","last_peered":"2023-09-30T00:17:51.975442+0000","last_clean":"2023-09-30T00:17:51.975442+0000","last_became_active":"2023-09-30T00:17:51.961348+0000","last_became_peered":"2023-09-30T00:17:51.961348+0000","last_unstale":"2023-09-30T00:17:51.975442+0000","last_undegraded":"2023-09-30T00:17:51.975442+0000","last_fullsized":"2023-09-30T00:17:51.975442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:11:09.140469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975871+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.975871+0000","last_peered":"2023-09-30T00:17:51.975871+0000","last_clean":"2023-09-30T00:17:51.975871+0000","last_became_active":"2023-09-30T00:17:51.963534+0000","last_became_peered":"2023-09-30T00:17:51.963534+0000","last_unstale":"2023-09-30T00:17:51.975871+0000","last_undegraded":"2023-09-30T00:17:51.975871+0000","last_fullsized":"2023-09-30T00:17:51.975871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:21:49.086269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.569843+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:31.844594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972810+0000","last_change":"2023-09-30T00:17:51.968803+0000","last_active":"2023-09-30T00:17:51.972810+0000","last_peered":"2023-09-30T00:17:51.972810+0000","last_clean":"2023-09-30T00:17:51.972810+0000","last_became_active":"2023-09-30T00:17:51.965833+0000","last_became_peered":"2023-09-30T00:17:51.965833+0000","last_unstale":"2023-09-30T00:17:51.972810+0000","last_undegraded":"2023-09-30T00:17:51.972810+0000","last_fullsized":"2023-09-30T00:17:51.972810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960606+0000","last_change":"2023-09-30T00:17:51.592497+0000","last_active":"2023-09-30T00:17:51.960606+0000","last_peered":"2023-09-30T00:17:51.960606+0000","last_clean":"2023-09-30T00:17:51.960606+0000","last_became_active":"2023-09-30T00:17:51.585128+0000","last_became_peered":"2023-09-30T00:17:51.585128+0000","last_unstale":"2023-09-30T00:17:51.960606+0000","last_undegraded":"2023-09-30T00:17:51.960606+0000","last_fullsized":"2023-09-30T00:17:51.960606+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:46.058014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977744+0000","last_change":"2023-09-30T00:17:51.974520+0000","last_active":"2023-09-30T00:17:51.977744+0000","last_peered":"2023-09-30T00:17:51.977744+0000","last_clean":"2023-09-30T00:17:51.977744+0000","last_became_active":"2023-09-30T00:17:51.974109+0000","last_became_peered":"2023-09-30T00:17:51.974109+0000","last_unstale":"2023-09-30T00:17:51.977744+0000","last_undegraded":"2023-09-30T00:17:51.977744+0000","last_fullsized":"2023-09-30T00:17:51.977744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:25:05.134336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941011+0000","last_change":"2023-09-30T00:17:51.587986+0000","last_active":"2023-09-30T00:17:51.941011+0000","last_peered":"2023-09-30T00:17:51.941011+0000","last_clean":"2023-09-30T00:17:51.941011+0000","last_became_active":"2023-09-30T00:17:51.587724+0000","last_became_peered":"2023-09-30T00:17:51.587724+0000","last_unstale":"2023-09-30T00:17:51.941011+0000","last_undegraded":"2023-09-30T00:17:51.941011+0000","last_fullsized":"2023-09-30T00:17:51.941011+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568369+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:56:13.933771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952700+0000","last_change":"2023-09-30T00:17:51.596511+0000","last_active":"2023-09-30T00:17:51.952700+0000","last_peered":"2023-09-30T00:17:51.952700+0000","last_clean":"2023-09-30T00:17:51.952700+0000","last_became_active":"2023-09-30T00:17:51.596034+0000","last_became_peered":"2023-09-30T00:17:51.596034+0000","last_unstale":"2023-09-30T00:17:51.952700+0000","last_undegraded":"2023-09-30T00:17:51.952700+0000","last_fullsized":"2023-09-30T00:17:51.952700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:44:29.988610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969658+0000","last_change":"2023-09-30T00:17:51.964766+0000","last_active":"2023-09-30T00:17:51.969658+0000","last_peered":"2023-09-30T00:17:51.969658+0000","last_clean":"2023-09-30T00:17:51.969658+0000","last_became_active":"2023-09-30T00:17:51.964496+0000","last_became_peered":"2023-09-30T00:17:51.964496+0000","last_unstale":"2023-09-30T00:17:51.969658+0000","last_undegraded":"2023-09-30T00:17:51.969658+0000","last_fullsized":"2023-09-30T00:17:51.969658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.942081+0000","last_change":"2023-09-30T00:17:51.577022+0000","last_active":"2023-09-30T00:17:51.942081+0000","last_peered":"2023-09-30T00:17:51.942081+0000","last_clean":"2023-09-30T00:17:51.942081+0000","last_became_active":"2023-09-30T00:17:51.576726+0000","last_became_peered":"2023-09-30T00:17:51.576726+0000","last_unstale":"2023-09-30T00:17:51.942081+0000","last_undegraded":"2023-09-30T00:17:51.942081+0000","last_fullsized":"2023-09-30T00:17:51.942081+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972739+0000","last_change":"2023-09-30T00:17:51.969564+0000","last_active":"2023-09-30T00:17:51.972739+0000","last_peered":"2023-09-30T00:17:51.972739+0000","last_clean":"2023-09-30T00:17:51.972739+0000","last_became_active":"2023-09-30T00:17:51.969240+0000","last_became_peered":"2023-09-30T00:17:51.969240+0000","last_unstale":"2023-09-30T00:17:51.972739+0000","last_undegraded":"2023-09-30T00:17:51.972739+0000","last_fullsized":"2023-09-30T00:17:51.972739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:42.701890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.589757+0000","last_change":"2023-09-30T00:17:51.588716+0000","last_active":"2023-09-30T00:17:51.589757+0000","last_peered":"2023-09-30T00:17:51.589757+0000","last_clean":"2023-09-30T00:17:51.589757+0000","last_became_active":"2023-09-30T00:17:51.588454+0000","last_became_peered":"2023-09-30T00:17:51.588454+0000","last_unstale":"2023-09-30T00:17:51.589757+0000","last_undegraded":"2023-09-30T00:17:51.589757+0000","last_fullsized":"2023-09-30T00:17:51.589757+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568152+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:48:09.502104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"18'3","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965653+0000","last_change":"2023-09-30T00:17:51.589988+0000","last_active":"2023-09-30T00:17:51.965653+0000","last_peered":"2023-09-30T00:17:51.965653+0000","last_clean":"2023-09-30T00:17:51.965653+0000","last_became_active":"2023-09-30T00:17:51.589729+0000","last_became_peered":"2023-09-30T00:17:51.589729+0000","last_unstale":"2023-09-30T00:17:51.965653+0000","last_undegraded":"2023-09-30T00:17:51.965653+0000","last_fullsized":"2023-09-30T00:17:51.965653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975239+0000","last_change":"2023-09-30T00:17:51.967245+0000","last_active":"2023-09-30T00:17:51.975239+0000","last_peered":"2023-09-30T00:17:51.975239+0000","last_clean":"2023-09-30T00:17:51.975239+0000","last_became_active":"2023-09-30T00:17:51.963755+0000","last_became_peered":"2023-09-30T00:17:51.963755+0000","last_unstale":"2023-09-30T00:17:51.975239+0000","last_undegraded":"2023-09-30T00:17:51.975239+0000","last_fullsized":"2023-09-30T00:17:51.975239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:57:31.735059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"18'6","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.983073+0000","last_change":"2023-09-30T00:17:51.592307+0000","last_active":"2023-09-30T00:17:51.983073+0000","last_peered":"2023-09-30T00:17:51.983073+0000","last_clean":"2023-09-30T00:17:51.983073+0000","last_became_active":"2023-09-30T00:17:51.591753+0000","last_became_peered":"2023-09-30T00:17:51.591753+0000","last_unstale":"2023-09-30T00:17:51.983073+0000","last_undegraded":"2023-09-30T00:17:51.983073+0000","last_fullsized":"2023-09-30T00:17:51.983073+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:14:46.193926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.967859+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.967859+0000","last_peered":"2023-09-30T00:17:51.967859+0000","last_clean":"2023-09-30T00:17:51.967859+0000","last_became_active":"2023-09-30T00:17:51.967118+0000","last_became_peered":"2023-09-30T00:17:51.967118+0000","last_unstale":"2023-09-30T00:17:51.967859+0000","last_undegraded":"2023-09-30T00:17:51.967859+0000","last_fullsized":"2023-09-30T00:17:51.967859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:50.694814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969722+0000","last_change":"2023-09-30T00:17:51.591516+0000","last_active":"2023-09-30T00:17:51.969722+0000","last_peered":"2023-09-30T00:17:51.969722+0000","last_clean":"2023-09-30T00:17:51.969722+0000","last_became_active":"2023-09-30T00:17:51.591206+0000","last_became_peered":"2023-09-30T00:17:51.591206+0000","last_unstale":"2023-09-30T00:17:51.969722+0000","last_undegraded":"2023-09-30T00:17:51.969722+0000","last_fullsized":"2023-09-30T00:17:51.969722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:22.807978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977754+0000","last_change":"2023-09-30T00:17:51.974628+0000","last_active":"2023-09-30T00:17:51.977754+0000","last_peered":"2023-09-30T00:17:51.977754+0000","last_clean":"2023-09-30T00:17:51.977754+0000","last_became_active":"2023-09-30T00:17:51.974276+0000","last_became_peered":"2023-09-30T00:17:51.974276+0000","last_unstale":"2023-09-30T00:17:51.977754+0000","last_undegraded":"2023-09-30T00:17:51.977754+0000","last_fullsized":"2023-09-30T00:17:51.977754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.590891+0000","last_change":"2023-09-30T00:17:51.589138+0000","last_active":"2023-09-30T00:17:51.590891+0000","last_peered":"2023-09-30T00:17:51.590891+0000","last_clean":"2023-09-30T00:17:51.590891+0000","last_became_active":"2023-09-30T00:17:51.588883+0000","last_became_peered":"2023-09-30T00:17:51.588883+0000","last_unstale":"2023-09-30T00:17:51.590891+0000","last_undegraded":"2023-09-30T00:17:51.590891+0000","last_fullsized":"2023-09-30T00:17:51.590891+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977625+0000","last_change":"2023-09-30T00:17:51.972690+0000","last_active":"2023-09-30T00:17:51.977625+0000","last_peered":"2023-09-30T00:17:51.977625+0000","last_clean":"2023-09-30T00:17:51.977625+0000","last_became_active":"2023-09-30T00:17:51.969872+0000","last_became_peered":"2023-09-30T00:17:51.969872+0000","last_unstale":"2023-09-30T00:17:51.977625+0000","last_undegraded":"2023-09-30T00:17:51.977625+0000","last_fullsized":"2023-09-30T00:17:51.977625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952325+0000","last_change":"2023-09-30T00:17:51.597410+0000","last_active":"2023-09-30T00:17:51.952325+0000","last_peered":"2023-09-30T00:17:51.952325+0000","last_clean":"2023-09-30T00:17:51.952325+0000","last_became_active":"2023-09-30T00:17:51.597168+0000","last_became_peered":"2023-09-30T00:17:51.597168+0000","last_unstale":"2023-09-30T00:17:51.952325+0000","last_undegraded":"2023-09-30T00:17:51.952325+0000","last_fullsized":"2023-09-30T00:17:51.952325+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:44:45.339994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973574+0000","last_change":"2023-09-30T00:17:51.969752+0000","last_active":"2023-09-30T00:17:51.973574+0000","last_peered":"2023-09-30T00:17:51.973574+0000","last_clean":"2023-09-30T00:17:51.973574+0000","last_became_active":"2023-09-30T00:17:51.968993+0000","last_became_peered":"2023-09-30T00:17:51.968993+0000","last_unstale":"2023-09-30T00:17:51.973574+0000","last_undegraded":"2023-09-30T00:17:51.973574+0000","last_fullsized":"2023-09-30T00:17:51.973574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"18'8","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974927+0000","last_change":"2023-09-30T00:17:51.592339+0000","last_active":"2023-09-30T00:17:51.974927+0000","last_peered":"2023-09-30T00:17:51.974927+0000","last_clean":"2023-09-30T00:17:51.974927+0000","last_became_active":"2023-09-30T00:17:51.592134+0000","last_became_peered":"2023-09-30T00:17:51.592134+0000","last_unstale":"2023-09-30T00:17:51.974927+0000","last_undegraded":"2023-09-30T00:17:51.974927+0000","last_fullsized":"2023-09-30T00:17:51.974927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.569711+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:42:10.868448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"18'1","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.944741+0000","last_change":"2023-09-30T00:17:51.577740+0000","last_active":"2023-09-30T00:17:51.944741+0000","last_peered":"2023-09-30T00:17:51.944741+0000","last_clean":"2023-09-30T00:17:51.944741+0000","last_became_active":"2023-09-30T00:17:51.577449+0000","last_became_peered":"2023-09-30T00:17:51.577449+0000","last_unstale":"2023-09-30T00:17:51.944741+0000","last_undegraded":"2023-09-30T00:17:51.944741+0000","last_fullsized":"2023-09-30T00:17:51.944741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977756+0000","last_change":"2023-09-30T00:17:51.974425+0000","last_active":"2023-09-30T00:17:51.977756+0000","last_peered":"2023-09-30T00:17:51.977756+0000","last_clean":"2023-09-30T00:17:51.977756+0000","last_became_active":"2023-09-30T00:17:51.974143+0000","last_became_peered":"2023-09-30T00:17:51.974143+0000","last_unstale":"2023-09-30T00:17:51.977756+0000","last_undegraded":"2023-09-30T00:17:51.977756+0000","last_fullsized":"2023-09-30T00:17:51.977756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:04:54.968664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"18'2","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968091+0000","last_change":"2023-09-30T00:17:51.578710+0000","last_active":"2023-09-30T00:17:51.968091+0000","last_peered":"2023-09-30T00:17:51.968091+0000","last_clean":"2023-09-30T00:17:51.968091+0000","last_became_active":"2023-09-30T00:17:51.578450+0000","last_became_peered":"2023-09-30T00:17:51.578450+0000","last_unstale":"2023-09-30T00:17:51.968091+0000","last_undegraded":"2023-09-30T00:17:51.968091+0000","last_fullsized":"2023-09-30T00:17:51.968091+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.958310+0000","last_change":"2023-09-30T00:17:51.951367+0000","last_active":"2023-09-30T00:17:51.958310+0000","last_peered":"2023-09-30T00:17:51.958310+0000","last_clean":"2023-09-30T00:17:51.958310+0000","last_became_active":"2023-09-30T00:17:51.951031+0000","last_became_peered":"2023-09-30T00:17:51.951031+0000","last_unstale":"2023-09-30T00:17:51.958310+0000","last_undegraded":"2023-09-30T00:17:51.958310+0000","last_fullsized":"2023-09-30T00:17:51.958310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976462+0000","last_change":"2023-09-30T00:17:51.594234+0000","last_active":"2023-09-30T00:17:51.976462+0000","last_peered":"2023-09-30T00:17:51.976462+0000","last_clean":"2023-09-30T00:17:51.976462+0000","last_became_active":"2023-09-30T00:17:51.594049+0000","last_became_peered":"2023-09-30T00:17:51.594049+0000","last_unstale":"2023-09-30T00:17:51.976462+0000","last_undegraded":"2023-09-30T00:17:51.976462+0000","last_fullsized":"2023-09-30T00:17:51.976462+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:53:48.206055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965298+0000","last_change":"2023-09-30T00:17:51.965298+0000","last_active":"2023-09-30T00:17:51.965298+0000","last_peered":"2023-09-30T00:17:51.965298+0000","last_clean":"2023-09-30T00:17:51.965298+0000","last_became_active":"2023-09-30T00:17:51.964965+0000","last_became_peered":"2023-09-30T00:17:51.964965+0000","last_unstale":"2023-09-30T00:17:51.965298+0000","last_undegraded":"2023-09-30T00:17:51.965298+0000","last_fullsized":"2023-09-30T00:17:51.965298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:33:20.583162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.955628+0000","last_change":"2023-09-30T00:17:51.592485+0000","last_active":"2023-09-30T00:17:51.955628+0000","last_peered":"2023-09-30T00:17:51.955628+0000","last_clean":"2023-09-30T00:17:51.955628+0000","last_became_active":"2023-09-30T00:17:51.592054+0000","last_became_peered":"2023-09-30T00:17:51.592054+0000","last_unstale":"2023-09-30T00:17:51.955628+0000","last_undegraded":"2023-09-30T00:17:51.955628+0000","last_fullsized":"2023-09-30T00:17:51.955628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:46:52.872573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952057+0000","last_change":"2023-09-30T00:17:51.952057+0000","last_active":"2023-09-30T00:17:51.952057+0000","last_peered":"2023-09-30T00:17:51.952057+0000","last_clean":"2023-09-30T00:17:51.952057+0000","last_became_active":"2023-09-30T00:17:51.951537+0000","last_became_peered":"2023-09-30T00:17:51.951537+0000","last_unstale":"2023-09-30T00:17:51.952057+0000","last_undegraded":"2023-09-30T00:17:51.952057+0000","last_fullsized":"2023-09-30T00:17:51.952057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:20:25.199612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.578071+0000","last_change":"2023-09-30T00:17:51.578071+0000","last_active":"2023-09-30T00:17:51.578071+0000","last_peered":"2023-09-30T00:17:51.578071+0000","last_clean":"2023-09-30T00:17:51.578071+0000","last_became_active":"2023-09-30T00:17:51.577808+0000","last_became_peered":"2023-09-30T00:17:51.577808+0000","last_unstale":"2023-09-30T00:17:51.578071+0000","last_undegraded":"2023-09-30T00:17:51.578071+0000","last_fullsized":"2023-09-30T00:17:51.578071+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:20.626316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976183+0000","last_change":"2023-09-30T00:17:51.970822+0000","last_active":"2023-09-30T00:17:51.976183+0000","last_peered":"2023-09-30T00:17:51.976183+0000","last_clean":"2023-09-30T00:17:51.976183+0000","last_became_active":"2023-09-30T00:17:51.970520+0000","last_became_peered":"2023-09-30T00:17:51.970520+0000","last_unstale":"2023-09-30T00:17:51.976183+0000","last_undegraded":"2023-09-30T00:17:51.976183+0000","last_fullsized":"2023-09-30T00:17:51.976183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:07.347359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952250+0000","last_change":"2023-09-30T00:17:51.595723+0000","last_active":"2023-09-30T00:17:51.952250+0000","last_peered":"2023-09-30T00:17:51.952250+0000","last_clean":"2023-09-30T00:17:51.952250+0000","last_became_active":"2023-09-30T00:17:51.595473+0000","last_became_peered":"2023-09-30T00:17:51.595473+0000","last_unstale":"2023-09-30T00:17:51.952250+0000","last_undegraded":"2023-09-30T00:17:51.952250+0000","last_fullsized":"2023-09-30T00:17:51.952250+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:06.343890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980343+0000","last_change":"2023-09-30T00:17:51.977010+0000","last_active":"2023-09-30T00:17:51.980343+0000","last_peered":"2023-09-30T00:17:51.980343+0000","last_clean":"2023-09-30T00:17:51.980343+0000","last_became_active":"2023-09-30T00:17:51.976744+0000","last_became_peered":"2023-09-30T00:17:51.976744+0000","last_unstale":"2023-09-30T00:17:51.980343+0000","last_undegraded":"2023-09-30T00:17:51.980343+0000","last_fullsized":"2023-09-30T00:17:51.980343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:02:14.797729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952109+0000","last_change":"2023-09-30T00:17:51.595305+0000","last_active":"2023-09-30T00:17:51.952109+0000","last_peered":"2023-09-30T00:17:51.952109+0000","last_clean":"2023-09-30T00:17:51.952109+0000","last_became_active":"2023-09-30T00:17:51.595042+0000","last_became_peered":"2023-09-30T00:17:51.595042+0000","last_unstale":"2023-09-30T00:17:51.952109+0000","last_undegraded":"2023-09-30T00:17:51.952109+0000","last_fullsized":"2023-09-30T00:17:51.952109+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:54.681232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965816+0000","last_change":"2023-09-30T00:17:51.965816+0000","last_active":"2023-09-30T00:17:51.965816+0000","last_peered":"2023-09-30T00:17:51.965816+0000","last_clean":"2023-09-30T00:17:51.965816+0000","last_became_active":"2023-09-30T00:17:51.965549+0000","last_became_peered":"2023-09-30T00:17:51.965549+0000","last_unstale":"2023-09-30T00:17:51.965816+0000","last_undegraded":"2023-09-30T00:17:51.965816+0000","last_fullsized":"2023-09-30T00:17:51.965816+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:30.526216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.953542+0000","last_change":"2023-09-30T00:17:51.596289+0000","last_active":"2023-09-30T00:17:51.953542+0000","last_peered":"2023-09-30T00:17:51.953542+0000","last_clean":"2023-09-30T00:17:51.953542+0000","last_became_active":"2023-09-30T00:17:51.595689+0000","last_became_peered":"2023-09-30T00:17:51.595689+0000","last_unstale":"2023-09-30T00:17:51.953542+0000","last_undegraded":"2023-09-30T00:17:51.953542+0000","last_fullsized":"2023-09-30T00:17:51.953542+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:46:55.016362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975889+0000","last_change":"2023-09-30T00:17:51.970420+0000","last_active":"2023-09-30T00:17:51.975889+0000","last_peered":"2023-09-30T00:17:51.975889+0000","last_clean":"2023-09-30T00:17:51.975889+0000","last_became_active":"2023-09-30T00:17:51.963999+0000","last_became_peered":"2023-09-30T00:17:51.963999+0000","last_unstale":"2023-09-30T00:17:51.975889+0000","last_undegraded":"2023-09-30T00:17:51.975889+0000","last_fullsized":"2023-09-30T00:17:51.975889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962989+0000","last_change":"2023-09-30T00:17:51.590485+0000","last_active":"2023-09-30T00:17:51.962989+0000","last_peered":"2023-09-30T00:17:51.962989+0000","last_clean":"2023-09-30T00:17:51.962989+0000","last_became_active":"2023-09-30T00:17:51.590210+0000","last_became_peered":"2023-09-30T00:17:51.590210+0000","last_unstale":"2023-09-30T00:17:51.962989+0000","last_undegraded":"2023-09-30T00:17:51.962989+0000","last_fullsized":"2023-09-30T00:17:51.962989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:32.708671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.582012+0000","last_change":"2023-09-30T00:17:45.196688+0000","last_active":"2023-09-30T00:17:51.582012+0000","last_peered":"2023-09-30T00:17:51.582012+0000","last_clean":"2023-09-30T00:17:51.582012+0000","last_became_active":"2023-09-30T00:17:45.196361+0000","last_became_peered":"2023-09-30T00:17:45.196361+0000","last_unstale":"2023-09-30T00:17:51.582012+0000","last_undegraded":"2023-09-30T00:17:51.582012+0000","last_fullsized":"2023-09-30T00:17:51.582012+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977735+0000","last_change":"2023-09-30T00:17:51.974386+0000","last_active":"2023-09-30T00:17:51.977735+0000","last_peered":"2023-09-30T00:17:51.977735+0000","last_clean":"2023-09-30T00:17:51.977735+0000","last_became_active":"2023-09-30T00:17:51.973776+0000","last_became_peered":"2023-09-30T00:17:51.973776+0000","last_unstale":"2023-09-30T00:17:51.977735+0000","last_undegraded":"2023-09-30T00:17:51.977735+0000","last_fullsized":"2023-09-30T00:17:51.977735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:25:21.282436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:17:50.566871+0000","last_change":"2023-09-30T00:17:45.203067+0000","last_active":"2023-09-30T00:17:50.566871+0000","last_peered":"2023-09-30T00:17:50.566871+0000","last_clean":"2023-09-30T00:17:50.566871+0000","last_became_active":"2023-09-30T00:17:45.202823+0000","last_became_peered":"2023-09-30T00:17:45.202823+0000","last_unstale":"2023-09-30T00:17:50.566871+0000","last_undegraded":"2023-09-30T00:17:50.566871+0000","last_fullsized":"2023-09-30T00:17:50.566871+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:26:01.195302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943027+0000","last_change":"2023-09-30T00:17:51.592843+0000","last_active":"2023-09-30T00:17:51.943027+0000","last_peered":"2023-09-30T00:17:51.943027+0000","last_clean":"2023-09-30T00:17:51.943027+0000","last_became_active":"2023-09-30T00:17:51.592601+0000","last_became_peered":"2023-09-30T00:17:51.592601+0000","last_unstale":"2023-09-30T00:17:51.943027+0000","last_undegraded":"2023-09-30T00:17:51.943027+0000","last_fullsized":"2023-09-30T00:17:51.943027+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965286+0000","last_change":"2023-09-30T00:17:51.965286+0000","last_active":"2023-09-30T00:17:51.965286+0000","last_peered":"2023-09-30T00:17:51.965286+0000","last_clean":"2023-09-30T00:17:51.965286+0000","last_became_active":"2023-09-30T00:17:51.964869+0000","last_became_peered":"2023-09-30T00:17:51.964869+0000","last_unstale":"2023-09-30T00:17:51.965286+0000","last_undegraded":"2023-09-30T00:17:51.965286+0000","last_fullsized":"2023-09-30T00:17:51.965286+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:46.192135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:17:50.566873+0000","last_change":"2023-09-30T00:17:45.195737+0000","last_active":"2023-09-30T00:17:50.566873+0000","last_peered":"2023-09-30T00:17:50.566873+0000","last_clean":"2023-09-30T00:17:50.566873+0000","last_became_active":"2023-09-30T00:17:45.195464+0000","last_became_peered":"2023-09-30T00:17:45.195464+0000","last_unstale":"2023-09-30T00:17:50.566873+0000","last_undegraded":"2023-09-30T00:17:50.566873+0000","last_fullsized":"2023-09-30T00:17:50.566873+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:24:48.730978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952759+0000","last_change":"2023-09-30T00:17:47.199132+0000","last_active":"2023-09-30T00:17:51.952759+0000","last_peered":"2023-09-30T00:17:51.952759+0000","last_clean":"2023-09-30T00:17:51.952759+0000","last_became_active":"2023-09-30T00:17:47.198255+0000","last_became_peered":"2023-09-30T00:17:47.198255+0000","last_unstale":"2023-09-30T00:17:51.952759+0000","last_undegraded":"2023-09-30T00:17:51.952759+0000","last_fullsized":"2023-09-30T00:17:51.952759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_clean_scrub_stamp":"2023-09-30T00:17:46.189460+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:09.834201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.575046+0000","last_change":"2023-09-30T00:17:51.575046+0000","last_active":"2023-09-30T00:17:51.575046+0000","last_peered":"2023-09-30T00:17:51.575046+0000","last_clean":"2023-09-30T00:17:51.575046+0000","last_became_active":"2023-09-30T00:17:51.574815+0000","last_became_peered":"2023-09-30T00:17:51.574815+0000","last_unstale":"2023-09-30T00:17:51.575046+0000","last_undegraded":"2023-09-30T00:17:51.575046+0000","last_fullsized":"2023-09-30T00:17:51.575046+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:43:57.892326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973520+0000","last_change":"2023-09-30T00:17:51.969276+0000","last_active":"2023-09-30T00:17:51.973520+0000","last_peered":"2023-09-30T00:17:51.973520+0000","last_clean":"2023-09-30T00:17:51.973520+0000","last_became_active":"2023-09-30T00:17:51.966513+0000","last_became_peered":"2023-09-30T00:17:51.966513+0000","last_unstale":"2023-09-30T00:17:51.973520+0000","last_undegraded":"2023-09-30T00:17:51.973520+0000","last_fullsized":"2023-09-30T00:17:51.973520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.579721+0000","last_change":"2023-09-30T00:17:45.197230+0000","last_active":"2023-09-30T00:17:51.579721+0000","last_peered":"2023-09-30T00:17:51.579721+0000","last_clean":"2023-09-30T00:17:51.579721+0000","last_became_active":"2023-09-30T00:17:45.196528+0000","last_became_peered":"2023-09-30T00:17:45.196528+0000","last_unstale":"2023-09-30T00:17:51.579721+0000","last_undegraded":"2023-09-30T00:17:51.579721+0000","last_fullsized":"2023-09-30T00:17:51.579721+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.575520+0000","last_change":"2023-09-30T00:17:51.575520+0000","last_active":"2023-09-30T00:17:51.575520+0000","last_peered":"2023-09-30T00:17:51.575520+0000","last_clean":"2023-09-30T00:17:51.575520+0000","last_became_active":"2023-09-30T00:17:51.575306+0000","last_became_peered":"2023-09-30T00:17:51.575306+0000","last_unstale":"2023-09-30T00:17:51.575520+0000","last_undegraded":"2023-09-30T00:17:51.575520+0000","last_fullsized":"2023-09-30T00:17:51.575520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:31:32.127076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568586+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:37:36.608579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.940896+0000","last_change":"2023-09-30T00:17:45.196542+0000","last_active":"2023-09-30T00:17:51.940896+0000","last_peered":"2023-09-30T00:17:51.940896+0000","last_clean":"2023-09-30T00:17:51.940896+0000","last_became_active":"2023-09-30T00:17:45.196123+0000","last_became_peered":"2023-09-30T00:17:45.196123+0000","last_unstale":"2023-09-30T00:17:51.940896+0000","last_undegraded":"2023-09-30T00:17:51.940896+0000","last_fullsized":"2023-09-30T00:17:51.940896+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952371+0000","last_change":"2023-09-30T00:17:51.591518+0000","last_active":"2023-09-30T00:17:51.952371+0000","last_peered":"2023-09-30T00:17:51.952371+0000","last_clean":"2023-09-30T00:17:51.952371+0000","last_became_active":"2023-09-30T00:17:51.591253+0000","last_became_peered":"2023-09-30T00:17:51.591253+0000","last_unstale":"2023-09-30T00:17:51.952371+0000","last_undegraded":"2023-09-30T00:17:51.952371+0000","last_fullsized":"2023-09-30T00:17:51.952371+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:52:46.330854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980323+0000","last_change":"2023-09-30T00:17:51.976800+0000","last_active":"2023-09-30T00:17:51.980323+0000","last_peered":"2023-09-30T00:17:51.980323+0000","last_clean":"2023-09-30T00:17:51.980323+0000","last_became_active":"2023-09-30T00:17:51.976602+0000","last_became_peered":"2023-09-30T00:17:51.976602+0000","last_unstale":"2023-09-30T00:17:51.980323+0000","last_undegraded":"2023-09-30T00:17:51.980323+0000","last_fullsized":"2023-09-30T00:17:51.980323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:53:26.550299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952216+0000","last_change":"2023-09-30T00:17:45.207123+0000","last_active":"2023-09-30T00:17:51.952216+0000","last_peered":"2023-09-30T00:17:51.952216+0000","last_clean":"2023-09-30T00:17:51.952216+0000","last_became_active":"2023-09-30T00:17:45.206750+0000","last_became_peered":"2023-09-30T00:17:45.206750+0000","last_unstale":"2023-09-30T00:17:51.952216+0000","last_undegraded":"2023-09-30T00:17:51.952216+0000","last_fullsized":"2023-09-30T00:17:51.952216+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.587209+0000","last_change":"2023-09-30T00:17:51.585515+0000","last_active":"2023-09-30T00:17:51.587209+0000","last_peered":"2023-09-30T00:17:51.587209+0000","last_clean":"2023-09-30T00:17:51.587209+0000","last_became_active":"2023-09-30T00:17:51.585262+0000","last_became_peered":"2023-09-30T00:17:51.585262+0000","last_unstale":"2023-09-30T00:17:51.587209+0000","last_undegraded":"2023-09-30T00:17:51.587209+0000","last_fullsized":"2023-09-30T00:17:51.587209+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:04.914282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568596+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:59.954235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.954829+0000","last_change":"2023-09-30T00:17:45.206971+0000","last_active":"2023-09-30T00:17:51.954829+0000","last_peered":"2023-09-30T00:17:51.954829+0000","last_clean":"2023-09-30T00:17:51.954829+0000","last_became_active":"2023-09-30T00:17:45.206693+0000","last_became_peered":"2023-09-30T00:17:45.206693+0000","last_unstale":"2023-09-30T00:17:51.954829+0000","last_undegraded":"2023-09-30T00:17:51.954829+0000","last_fullsized":"2023-09-30T00:17:51.954829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.575506+0000","last_change":"2023-09-30T00:17:51.575506+0000","last_active":"2023-09-30T00:17:51.575506+0000","last_peered":"2023-09-30T00:17:51.575506+0000","last_clean":"2023-09-30T00:17:51.575506+0000","last_became_active":"2023-09-30T00:17:51.574806+0000","last_became_peered":"2023-09-30T00:17:51.574806+0000","last_unstale":"2023-09-30T00:17:51.575506+0000","last_undegraded":"2023-09-30T00:17:51.575506+0000","last_fullsized":"2023-09-30T00:17:51.575506+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:51.816991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975408+0000","last_change":"2023-09-30T00:17:51.967785+0000","last_active":"2023-09-30T00:17:51.975408+0000","last_peered":"2023-09-30T00:17:51.975408+0000","last_clean":"2023-09-30T00:17:51.975408+0000","last_became_active":"2023-09-30T00:17:51.967419+0000","last_became_peered":"2023-09-30T00:17:51.967419+0000","last_unstale":"2023-09-30T00:17:51.975408+0000","last_undegraded":"2023-09-30T00:17:51.975408+0000","last_fullsized":"2023-09-30T00:17:51.975408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941618+0000","last_change":"2023-09-30T00:17:45.203202+0000","last_active":"2023-09-30T00:17:51.941618+0000","last_peered":"2023-09-30T00:17:51.941618+0000","last_clean":"2023-09-30T00:17:51.941618+0000","last_became_active":"2023-09-30T00:17:45.202879+0000","last_became_peered":"2023-09-30T00:17:45.202879+0000","last_unstale":"2023-09-30T00:17:51.941618+0000","last_undegraded":"2023-09-30T00:17:51.941618+0000","last_fullsized":"2023-09-30T00:17:51.941618+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:50:42.379943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.585090+0000","last_change":"2023-09-30T00:17:51.580771+0000","last_active":"2023-09-30T00:17:51.585090+0000","last_peered":"2023-09-30T00:17:51.585090+0000","last_clean":"2023-09-30T00:17:51.585090+0000","last_became_active":"2023-09-30T00:17:51.580495+0000","last_became_peered":"2023-09-30T00:17:51.580495+0000","last_unstale":"2023-09-30T00:17:51.585090+0000","last_undegraded":"2023-09-30T00:17:51.585090+0000","last_fullsized":"2023-09-30T00:17:51.585090+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971832+0000","last_change":"2023-09-30T00:17:51.957745+0000","last_active":"2023-09-30T00:17:51.971832+0000","last_peered":"2023-09-30T00:17:51.971832+0000","last_clean":"2023-09-30T00:17:51.971832+0000","last_became_active":"2023-09-30T00:17:51.955331+0000","last_became_peered":"2023-09-30T00:17:51.955331+0000","last_unstale":"2023-09-30T00:17:51.971832+0000","last_undegraded":"2023-09-30T00:17:51.971832+0000","last_fullsized":"2023-09-30T00:17:51.971832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:45.393660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973877+0000","last_change":"2023-09-30T00:17:51.596021+0000","last_active":"2023-09-30T00:17:51.973877+0000","last_peered":"2023-09-30T00:17:51.973877+0000","last_clean":"2023-09-30T00:17:51.973877+0000","last_became_active":"2023-09-30T00:17:51.595745+0000","last_became_peered":"2023-09-30T00:17:51.595745+0000","last_unstale":"2023-09-30T00:17:51.973877+0000","last_undegraded":"2023-09-30T00:17:51.973877+0000","last_fullsized":"2023-09-30T00:17:51.973877+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:44.390191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.569794+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:06:14.351937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974567+0000","last_change":"2023-09-30T00:17:51.587276+0000","last_active":"2023-09-30T00:17:51.974567+0000","last_peered":"2023-09-30T00:17:51.974567+0000","last_clean":"2023-09-30T00:17:51.974567+0000","last_became_active":"2023-09-30T00:17:51.586984+0000","last_became_peered":"2023-09-30T00:17:51.586984+0000","last_unstale":"2023-09-30T00:17:51.974567+0000","last_undegraded":"2023-09-30T00:17:51.974567+0000","last_fullsized":"2023-09-30T00:17:51.974567+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.953787+0000","last_change":"2023-09-30T00:17:51.953787+0000","last_active":"2023-09-30T00:17:51.953787+0000","last_peered":"2023-09-30T00:17:51.953787+0000","last_clean":"2023-09-30T00:17:51.953787+0000","last_became_active":"2023-09-30T00:17:51.951805+0000","last_became_peered":"2023-09-30T00:17:51.951805+0000","last_unstale":"2023-09-30T00:17:51.953787+0000","last_undegraded":"2023-09-30T00:17:51.953787+0000","last_fullsized":"2023-09-30T00:17:51.953787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:57:04.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965610+0000","last_change":"2023-09-30T00:17:51.593225+0000","last_active":"2023-09-30T00:17:51.965610+0000","last_peered":"2023-09-30T00:17:51.965610+0000","last_clean":"2023-09-30T00:17:51.965610+0000","last_became_active":"2023-09-30T00:17:51.592913+0000","last_became_peered":"2023-09-30T00:17:51.592913+0000","last_unstale":"2023-09-30T00:17:51.965610+0000","last_undegraded":"2023-09-30T00:17:51.965610+0000","last_fullsized":"2023-09-30T00:17:51.965610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:37:56.776343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.955058+0000","last_change":"2023-09-30T00:17:51.955058+0000","last_active":"2023-09-30T00:17:51.955058+0000","last_peered":"2023-09-30T00:17:51.955058+0000","last_clean":"2023-09-30T00:17:51.955058+0000","last_became_active":"2023-09-30T00:17:51.954787+0000","last_became_peered":"2023-09-30T00:17:51.954787+0000","last_unstale":"2023-09-30T00:17:51.955058+0000","last_undegraded":"2023-09-30T00:17:51.955058+0000","last_fullsized":"2023-09-30T00:17:51.955058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:49:30.148218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.586229+0000","last_change":"2023-09-30T00:17:51.586229+0000","last_active":"2023-09-30T00:17:51.586229+0000","last_peered":"2023-09-30T00:17:51.586229+0000","last_clean":"2023-09-30T00:17:51.586229+0000","last_became_active":"2023-09-30T00:17:51.586027+0000","last_became_peered":"2023-09-30T00:17:51.586027+0000","last_unstale":"2023-09-30T00:17:51.586229+0000","last_undegraded":"2023-09-30T00:17:51.586229+0000","last_fullsized":"2023-09-30T00:17:51.586229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:48:13.817514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962963+0000","last_change":"2023-09-30T00:17:51.959628+0000","last_active":"2023-09-30T00:17:51.962963+0000","last_peered":"2023-09-30T00:17:51.962963+0000","last_clean":"2023-09-30T00:17:51.962963+0000","last_became_active":"2023-09-30T00:17:51.958974+0000","last_became_peered":"2023-09-30T00:17:51.958974+0000","last_unstale":"2023-09-30T00:17:51.962963+0000","last_undegraded":"2023-09-30T00:17:51.962963+0000","last_fullsized":"2023-09-30T00:17:51.962963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:28.604134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"18'4","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960480+0000","last_change":"2023-09-30T00:17:51.590387+0000","last_active":"2023-09-30T00:17:51.960480+0000","last_peered":"2023-09-30T00:17:51.960480+0000","last_clean":"2023-09-30T00:17:51.960480+0000","last_became_active":"2023-09-30T00:17:51.585479+0000","last_became_peered":"2023-09-30T00:17:51.585479+0000","last_unstale":"2023-09-30T00:17:51.960480+0000","last_undegraded":"2023-09-30T00:17:51.960480+0000","last_fullsized":"2023-09-30T00:17:51.960480+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.949418+0000","last_change":"2023-09-30T00:17:51.949418+0000","last_active":"2023-09-30T00:17:51.949418+0000","last_peered":"2023-09-30T00:17:51.949418+0000","last_clean":"2023-09-30T00:17:51.949418+0000","last_became_active":"2023-09-30T00:17:51.949101+0000","last_became_peered":"2023-09-30T00:17:51.949101+0000","last_unstale":"2023-09-30T00:17:51.949418+0000","last_undegraded":"2023-09-30T00:17:51.949418+0000","last_fullsized":"2023-09-30T00:17:51.949418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975274+0000","last_change":"2023-09-30T00:17:51.964155+0000","last_active":"2023-09-30T00:17:51.975274+0000","last_peered":"2023-09-30T00:17:51.975274+0000","last_clean":"2023-09-30T00:17:51.975274+0000","last_became_active":"2023-09-30T00:17:51.958334+0000","last_became_peered":"2023-09-30T00:17:51.958334+0000","last_unstale":"2023-09-30T00:17:51.975274+0000","last_undegraded":"2023-09-30T00:17:51.975274+0000","last_fullsized":"2023-09-30T00:17:51.975274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570057+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:31:59.960264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976104+0000","last_change":"2023-09-30T00:17:51.970791+0000","last_active":"2023-09-30T00:17:51.976104+0000","last_peered":"2023-09-30T00:17:51.976104+0000","last_clean":"2023-09-30T00:17:51.976104+0000","last_became_active":"2023-09-30T00:17:51.964142+0000","last_became_peered":"2023-09-30T00:17:51.964142+0000","last_unstale":"2023-09-30T00:17:51.976104+0000","last_undegraded":"2023-09-30T00:17:51.976104+0000","last_fullsized":"2023-09-30T00:17:51.976104+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568144+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:18:28.873560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570896+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:02:47.475511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672965,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9220,"kb_used_data":764,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362620,"statfs":{"total":96636764160,"available":96627322880,"internally_reserved":0,"allocated":782336,"data_stored":646622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]}]},{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8060,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363780,"statfs":{"total":96636764160,"available":96628510720,"internally_reserved":0,"allocated":184320,"data_stored":55478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8852,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94362988,"statfs":{"total":96636764160,"available":96627699712,"internally_reserved":0,"allocated":733184,"data_stored":634680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8052,"kb_used_data":172,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363788,"statfs":{"total":96636764160,"available":96628518912,"internally_reserved":0,"allocated":176128,"data_stored":55170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.859}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7779999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:17:55.983 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:17:56.128 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.128+0000 7f33d289f700 1 Processor -- start 2023-09-30T00:17:56.128 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.128+0000 7f33d289f700 1 -- start start 2023-09-30T00:17:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.128+0000 7f33d289f700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f33cc070c50 0x7f33cc0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.128+0000 7f33d289f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc0b7160 0x7f33cc0a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.128+0000 7f33d289f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0a1310 con 0x7f33cc13fd70 2023-09-30T00:17:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.128+0000 7f33d289f700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33cc071020 con 0x7f33cc070c50 2023-09-30T00:17:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.128+0000 7f33d289f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0b76a0 con 0x7f33cc0b7160 2023-09-30T00:17:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.128+0000 7f33cbfff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f33cc070c50 0x7f33cc0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.128+0000 7f33cb7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc0b7160 0x7f33cc0a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33cbfff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f33cc070c50 0x7f33cc0a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50518/0 (socket says 172.21.15.94:50518) 2023-09-30T00:17:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.128+0000 7f33d0e3c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f33cc13fd70 0x7f33cc0b4dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50198/0 (socket says 172.21.15.94:50198) 2023-09-30T00:17:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33cb7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc0b7160 0x7f33cc0a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39024/0 (socket says 172.21.15.94:39024) 2023-09-30T00:17:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33cbfff700 1 -- 172.21.15.94:0/947558120 learned_addr learned my addr 172.21.15.94:0/947558120 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33cbfff700 1 -- 172.21.15.94:0/947558120 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc0b7160 msgr2=0x7f33cc0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33cbfff700 1 --2- 172.21.15.94:0/947558120 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc0b7160 0x7f33cc0a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33cbfff700 1 -- 172.21.15.94:0/947558120 >> v1:172.21.15.94:6789/0 conn(0x7f33cc13fd70 legacy=0x7f33cc0b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33cbfff700 1 -- 172.21.15.94:0/947558120 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0b7940 con 0x7f33cc070c50 2023-09-30T00:17:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33caffd700 1 -- 172.21.15.94:0/947558120 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6983647 0 0) 0x7f33c4001490 con 0x7f33cc13fd70 2023-09-30T00:17:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33cb7fe700 1 --2- 172.21.15.94:0/947558120 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc0b7160 0x7f33cc0a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33cbfff700 1 --2- 172.21.15.94:0/947558120 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33cc070c50 0x7f33cc0a0dd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0099b0 tx=0x7f33bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c6d4543423b8aa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33caffd700 1 -- 172.21.15.94:0/947558120 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc009ea0 con 0x7f33cc070c50 2023-09-30T00:17:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33caffd700 1 -- 172.21.15.94:0/947558120 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc00b8f0 con 0x7f33cc070c50 2023-09-30T00:17:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.129+0000 7f33caffd700 1 -- 172.21.15.94:0/947558120 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc00bc90 con 0x7f33cc070c50 2023-09-30T00:17:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.130+0000 7f33d289f700 1 -- 172.21.15.94:0/947558120 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33cc070c50 msgr2=0x7f33cc0a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.130+0000 7f33d289f700 1 --2- 172.21.15.94:0/947558120 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33cc070c50 0x7f33cc0a0dd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0099b0 tx=0x7f33bc009250 comp rx=0 tx=0).stop 2023-09-30T00:17:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.130+0000 7f33d289f700 1 -- 172.21.15.94:0/947558120 shutdown_connections 2023-09-30T00:17:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.130+0000 7f33d289f700 1 --2- 172.21.15.94:0/947558120 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33cc070c50 0x7f33cc0a0dd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.130+0000 7f33d289f700 1 --2- 172.21.15.94:0/947558120 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc0b7160 0x7f33cc0a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.130+0000 7f33d289f700 1 -- 172.21.15.94:0/947558120 >> 172.21.15.94:0/947558120 conn(0x7f33cc06a090 msgr2=0x7f33cc0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:56.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.130+0000 7f33d289f700 1 -- 172.21.15.94:0/947558120 shutdown_connections 2023-09-30T00:17:56.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.130+0000 7f33d289f700 1 -- 172.21.15.94:0/947558120 wait complete. 2023-09-30T00:17:56.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.130+0000 7f33d289f700 1 Processor -- start 2023-09-30T00:17:56.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.131+0000 7f33d289f700 1 -- start start 2023-09-30T00:17:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.131+0000 7f33d289f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc070c50 0x7f33cc0ad370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.131+0000 7f33d289f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33cc0b7160 0x7f33cc1509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.131+0000 7f33d289f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33cc13fd70 0x7f33cc150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.131+0000 7f33d289f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc151430 con 0x7f33cc0b7160 2023-09-30T00:17:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.131+0000 7f33d289f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0adda0 con 0x7f33cc13fd70 2023-09-30T00:17:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.131+0000 7f33d289f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0acb90 con 0x7f33cc070c50 2023-09-30T00:17:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.131+0000 7f33cbfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc070c50 0x7f33cc0ad370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.131+0000 7f33cb7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33cc0b7160 0x7f33cc1509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.131+0000 7f33cb7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33cc0b7160 0x7f33cc1509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50526/0 (socket says 172.21.15.94:50526) 2023-09-30T00:17:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.131+0000 7f33cbfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc070c50 0x7f33cc0ad370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39026/0 (socket says 172.21.15.94:39026) 2023-09-30T00:17:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.131+0000 7f33cb7fe700 1 -- 172.21.15.94:0/2206513746 learned_addr learned my addr 172.21.15.94:0/2206513746 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.132+0000 7f33cbfff700 1 -- 172.21.15.94:0/2206513746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33cc13fd70 msgr2=0x7f33cc150ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.132+0000 7f33cbfff700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33cc13fd70 0x7f33cc150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.132+0000 7f33cbfff700 1 -- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33cc0b7160 msgr2=0x7f33cc1509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.132+0000 7f33cbfff700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33cc0b7160 0x7f33cc1509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.132+0000 7f33cbfff700 1 -- 172.21.15.94:0/2206513746 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0009420 con 0x7f33cc070c50 2023-09-30T00:17:56.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.132+0000 7f33cb7fe700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33cc0b7160 0x7f33cc1509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:56.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.132+0000 7f33cbfff700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc070c50 0x7f33cc0ad370 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f33cc067c90 tx=0x7f33bc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:56.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.132+0000 7f33b3fff700 1 -- 172.21.15.94:0/2206513746 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc00ba60 con 0x7f33cc070c50 2023-09-30T00:17:56.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.132+0000 7f33b3fff700 1 -- 172.21.15.94:0/2206513746 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc00bbc0 con 0x7f33cc070c50 2023-09-30T00:17:56.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.132+0000 7f33b3fff700 1 -- 172.21.15.94:0/2206513746 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc026680 con 0x7f33cc070c50 2023-09-30T00:17:56.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.132+0000 7f33d289f700 1 -- 172.21.15.94:0/2206513746 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33bc0094f0 con 0x7f33cc070c50 2023-09-30T00:17:56.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.132+0000 7f33d289f700 1 -- 172.21.15.94:0/2206513746 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc0ace70 con 0x7f33cc070c50 2023-09-30T00:17:56.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.133+0000 7f33d289f700 1 -- 172.21.15.94:0/2206513746 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33cc13e4c0 con 0x7f33cc070c50 2023-09-30T00:17:56.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.134+0000 7f33b3fff700 1 -- 172.21.15.94:0/2206513746 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f33bc023030 con 0x7f33cc070c50 2023-09-30T00:17:56.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.134+0000 7f33b3fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:56.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.134+0000 7f33b3fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:56.139 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.134+0000 7f33b3fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:56.139 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.134+0000 7f33b3fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:56.139 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.134+0000 7f33b3fff700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f33b8098700 0x7f33b809abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.139 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.134+0000 7f33cb7fe700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f33b8098700 0x7f33b809abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.139 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.134+0000 7f33b3fff700 1 -- 172.21.15.94:0/2206513746 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f33bc0c1790 con 0x7f33cc070c50 2023-09-30T00:17:56.139 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.135+0000 7f33cb7fe700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f33b8098700 0x7f33b809abb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0704d0 tx=0x7f33c00091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:56.139 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.138+0000 7f33b3fff700 1 -- 172.21.15.94:0/2206513746 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f33bc08eed0 con 0x7f33cc070c50 2023-09-30T00:17:56.304 INFO:tasks.ceph.mds.a.smithi094.stderr:2023-09-30T00:17:56.303+0000 7f5305eef700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-09-30T00:17:56.318 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.318+0000 7f33d289f700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-30T00:17:56.319 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.318+0000 7f33d289f700 1 -- 172.21.15.94:0/2206513746 --> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33cc0a0930 con 0x7f33b8098700 2023-09-30T00:17:56.325 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.325+0000 7f33b3fff700 1 -- 172.21.15.94:0/2206513746 <== mgr.4106 v2:172.21.15.94:6808/84242 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+268834 (secure 0 0 0) 0x7f33cc0a0930 con 0x7f33b8098700 2023-09-30T00:17:56.325 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.325+0000 7f33b3fff700 20 mgrc handle_command_reply tid 0 r 0 2023-09-30T00:17:56.328 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:56.330 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 10 mgrc shutdown 2023-09-30T00:17:56.331 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 1 -- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f33b8098700 msgr2=0x7f33b809abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.331 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f33b8098700 0x7f33b809abb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0704d0 tx=0x7f33c00091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.331 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 1 -- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc070c50 msgr2=0x7f33cc0ad370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.331 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc070c50 0x7f33cc0ad370 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f33cc067c90 tx=0x7f33bc017000 comp rx=0 tx=0).stop 2023-09-30T00:17:56.331 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 1 -- 172.21.15.94:0/2206513746 shutdown_connections 2023-09-30T00:17:56.331 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f33b8098700 0x7f33b809abb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.331 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f33cc070c50 0x7f33cc0ad370 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.331 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33cc13fd70 0x7f33cc150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.331 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 1 --2- 172.21.15.94:0/2206513746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33cc0b7160 0x7f33cc1509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.331 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 1 -- 172.21.15.94:0/2206513746 >> 172.21.15.94:0/2206513746 conn(0x7f33cc06a090 msgr2=0x7f33cc0b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:56.331 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 1 -- 172.21.15.94:0/2206513746 shutdown_connections 2023-09-30T00:17:56.332 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.328+0000 7f33d289f700 1 -- 172.21.15.94:0/2206513746 wait complete. 2023-09-30T00:17:56.332 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2023-09-30T00:17:56.340 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-09-30T00:17:55.851278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":202,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":34184,"kb_used_data":1832,"kb_used_omap":0,"kb_used_meta":32320,"kb_avail":377453176,"statfs":{"total":386547056640,"available":386512052224,"internally_reserved":0,"allocated":1875968,"data_stored":1391950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33095680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.286619"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960193+0000","last_change":"2023-09-30T00:17:51.960193+0000","last_active":"2023-09-30T00:17:51.960193+0000","last_peered":"2023-09-30T00:17:51.960193+0000","last_clean":"2023-09-30T00:17:51.960193+0000","last_became_active":"2023-09-30T00:17:51.959888+0000","last_became_peered":"2023-09-30T00:17:51.959888+0000","last_unstale":"2023-09-30T00:17:51.960193+0000","last_undegraded":"2023-09-30T00:17:51.960193+0000","last_fullsized":"2023-09-30T00:17:51.960193+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:40:42.458928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968577+0000","last_change":"2023-09-30T00:17:51.968577+0000","last_active":"2023-09-30T00:17:51.968577+0000","last_peered":"2023-09-30T00:17:51.968577+0000","last_clean":"2023-09-30T00:17:51.968577+0000","last_became_active":"2023-09-30T00:17:51.968275+0000","last_became_peered":"2023-09-30T00:17:51.968275+0000","last_unstale":"2023-09-30T00:17:51.968577+0000","last_undegraded":"2023-09-30T00:17:51.968577+0000","last_fullsized":"2023-09-30T00:17:51.968577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:19:24.974156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977703+0000","last_change":"2023-09-30T00:17:51.974359+0000","last_active":"2023-09-30T00:17:51.977703+0000","last_peered":"2023-09-30T00:17:51.977703+0000","last_clean":"2023-09-30T00:17:51.977703+0000","last_became_active":"2023-09-30T00:17:51.973789+0000","last_became_peered":"2023-09-30T00:17:51.973789+0000","last_unstale":"2023-09-30T00:17:51.977703+0000","last_undegraded":"2023-09-30T00:17:51.977703+0000","last_fullsized":"2023-09-30T00:17:51.977703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:27.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971859+0000","last_change":"2023-09-30T00:17:51.958127+0000","last_active":"2023-09-30T00:17:51.971859+0000","last_peered":"2023-09-30T00:17:51.971859+0000","last_clean":"2023-09-30T00:17:51.971859+0000","last_became_active":"2023-09-30T00:17:51.957850+0000","last_became_peered":"2023-09-30T00:17:51.957850+0000","last_unstale":"2023-09-30T00:17:51.971859+0000","last_undegraded":"2023-09-30T00:17:51.971859+0000","last_fullsized":"2023-09-30T00:17:51.971859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:48:36.950659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973474+0000","last_change":"2023-09-30T00:17:51.966792+0000","last_active":"2023-09-30T00:17:51.973474+0000","last_peered":"2023-09-30T00:17:51.973474+0000","last_clean":"2023-09-30T00:17:51.973474+0000","last_became_active":"2023-09-30T00:17:51.965887+0000","last_became_peered":"2023-09-30T00:17:51.965887+0000","last_unstale":"2023-09-30T00:17:51.973474+0000","last_undegraded":"2023-09-30T00:17:51.973474+0000","last_fullsized":"2023-09-30T00:17:51.973474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973636+0000","last_change":"2023-09-30T00:17:51.967175+0000","last_active":"2023-09-30T00:17:51.973636+0000","last_peered":"2023-09-30T00:17:51.973636+0000","last_clean":"2023-09-30T00:17:51.973636+0000","last_became_active":"2023-09-30T00:17:51.966408+0000","last_became_peered":"2023-09-30T00:17:51.966408+0000","last_unstale":"2023-09-30T00:17:51.973636+0000","last_undegraded":"2023-09-30T00:17:51.973636+0000","last_fullsized":"2023-09-30T00:17:51.973636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:47:56.029102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972783+0000","last_change":"2023-09-30T00:17:51.964264+0000","last_active":"2023-09-30T00:17:51.972783+0000","last_peered":"2023-09-30T00:17:51.972783+0000","last_clean":"2023-09-30T00:17:51.972783+0000","last_became_active":"2023-09-30T00:17:51.962740+0000","last_became_peered":"2023-09-30T00:17:51.962740+0000","last_unstale":"2023-09-30T00:17:51.972783+0000","last_undegraded":"2023-09-30T00:17:51.972783+0000","last_fullsized":"2023-09-30T00:17:51.972783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:52:26.654463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975296+0000","last_change":"2023-09-30T00:17:51.967599+0000","last_active":"2023-09-30T00:17:51.975296+0000","last_peered":"2023-09-30T00:17:51.975296+0000","last_clean":"2023-09-30T00:17:51.975296+0000","last_became_active":"2023-09-30T00:17:51.961762+0000","last_became_peered":"2023-09-30T00:17:51.961762+0000","last_unstale":"2023-09-30T00:17:51.975296+0000","last_undegraded":"2023-09-30T00:17:51.975296+0000","last_fullsized":"2023-09-30T00:17:51.975296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:35:31.620523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570714+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:14:31.363561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972400+0000","last_change":"2023-09-30T00:17:51.963624+0000","last_active":"2023-09-30T00:17:51.972400+0000","last_peered":"2023-09-30T00:17:51.972400+0000","last_clean":"2023-09-30T00:17:51.972400+0000","last_became_active":"2023-09-30T00:17:51.963303+0000","last_became_peered":"2023-09-30T00:17:51.963303+0000","last_unstale":"2023-09-30T00:17:51.972400+0000","last_undegraded":"2023-09-30T00:17:51.972400+0000","last_fullsized":"2023-09-30T00:17:51.972400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971954+0000","last_change":"2023-09-30T00:17:51.961620+0000","last_active":"2023-09-30T00:17:51.971954+0000","last_peered":"2023-09-30T00:17:51.971954+0000","last_clean":"2023-09-30T00:17:51.971954+0000","last_became_active":"2023-09-30T00:17:51.958237+0000","last_became_peered":"2023-09-30T00:17:51.958237+0000","last_unstale":"2023-09-30T00:17:51.971954+0000","last_undegraded":"2023-09-30T00:17:51.971954+0000","last_fullsized":"2023-09-30T00:17:51.971954+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974397+0000","last_change":"2023-09-30T00:17:51.974397+0000","last_active":"2023-09-30T00:17:51.974397+0000","last_peered":"2023-09-30T00:17:51.974397+0000","last_clean":"2023-09-30T00:17:51.974397+0000","last_became_active":"2023-09-30T00:17:51.974025+0000","last_became_peered":"2023-09-30T00:17:51.974025+0000","last_unstale":"2023-09-30T00:17:51.974397+0000","last_undegraded":"2023-09-30T00:17:51.974397+0000","last_fullsized":"2023-09-30T00:17:51.974397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:49:07.723751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570140+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:39:04.507564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976147+0000","last_change":"2023-09-30T00:17:51.970985+0000","last_active":"2023-09-30T00:17:51.976147+0000","last_peered":"2023-09-30T00:17:51.976147+0000","last_clean":"2023-09-30T00:17:51.976147+0000","last_became_active":"2023-09-30T00:17:51.966979+0000","last_became_peered":"2023-09-30T00:17:51.966979+0000","last_unstale":"2023-09-30T00:17:51.976147+0000","last_undegraded":"2023-09-30T00:17:51.976147+0000","last_fullsized":"2023-09-30T00:17:51.976147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:09:28.833739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975911+0000","last_change":"2023-09-30T00:17:51.970398+0000","last_active":"2023-09-30T00:17:51.975911+0000","last_peered":"2023-09-30T00:17:51.975911+0000","last_clean":"2023-09-30T00:17:51.975911+0000","last_became_active":"2023-09-30T00:17:51.961675+0000","last_became_peered":"2023-09-30T00:17:51.961675+0000","last_unstale":"2023-09-30T00:17:51.975911+0000","last_undegraded":"2023-09-30T00:17:51.975911+0000","last_fullsized":"2023-09-30T00:17:51.975911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:33:21.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972718+0000","last_change":"2023-09-30T00:17:51.969081+0000","last_active":"2023-09-30T00:17:51.972718+0000","last_peered":"2023-09-30T00:17:51.972718+0000","last_clean":"2023-09-30T00:17:51.972718+0000","last_became_active":"2023-09-30T00:17:51.966447+0000","last_became_peered":"2023-09-30T00:17:51.966447+0000","last_unstale":"2023-09-30T00:17:51.972718+0000","last_undegraded":"2023-09-30T00:17:51.972718+0000","last_fullsized":"2023-09-30T00:17:51.972718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:19:12.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960350+0000","last_change":"2023-09-30T00:17:51.953980+0000","last_active":"2023-09-30T00:17:51.960350+0000","last_peered":"2023-09-30T00:17:51.960350+0000","last_clean":"2023-09-30T00:17:51.960350+0000","last_became_active":"2023-09-30T00:17:51.952191+0000","last_became_peered":"2023-09-30T00:17:51.952191+0000","last_unstale":"2023-09-30T00:17:51.960350+0000","last_undegraded":"2023-09-30T00:17:51.960350+0000","last_fullsized":"2023-09-30T00:17:51.960350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:52:46.774408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973554+0000","last_change":"2023-09-30T00:17:51.969589+0000","last_active":"2023-09-30T00:17:51.973554+0000","last_peered":"2023-09-30T00:17:51.973554+0000","last_clean":"2023-09-30T00:17:51.973554+0000","last_became_active":"2023-09-30T00:17:51.967300+0000","last_became_peered":"2023-09-30T00:17:51.967300+0000","last_unstale":"2023-09-30T00:17:51.973554+0000","last_undegraded":"2023-09-30T00:17:51.973554+0000","last_fullsized":"2023-09-30T00:17:51.973554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:22:57.623135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568987+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:41:47.152317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972047+0000","last_change":"2023-09-30T00:17:51.961502+0000","last_active":"2023-09-30T00:17:51.972047+0000","last_peered":"2023-09-30T00:17:51.972047+0000","last_clean":"2023-09-30T00:17:51.972047+0000","last_became_active":"2023-09-30T00:17:51.961191+0000","last_became_peered":"2023-09-30T00:17:51.961191+0000","last_unstale":"2023-09-30T00:17:51.972047+0000","last_undegraded":"2023-09-30T00:17:51.972047+0000","last_fullsized":"2023-09-30T00:17:51.972047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:05:37.196886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976121+0000","last_change":"2023-09-30T00:17:51.970789+0000","last_active":"2023-09-30T00:17:51.976121+0000","last_peered":"2023-09-30T00:17:51.976121+0000","last_clean":"2023-09-30T00:17:51.976121+0000","last_became_active":"2023-09-30T00:17:51.967616+0000","last_became_peered":"2023-09-30T00:17:51.967616+0000","last_unstale":"2023-09-30T00:17:51.976121+0000","last_undegraded":"2023-09-30T00:17:51.976121+0000","last_fullsized":"2023-09-30T00:17:51.976121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976127+0000","last_change":"2023-09-30T00:17:51.970797+0000","last_active":"2023-09-30T00:17:51.976127+0000","last_peered":"2023-09-30T00:17:51.976127+0000","last_clean":"2023-09-30T00:17:51.976127+0000","last_became_active":"2023-09-30T00:17:51.963382+0000","last_became_peered":"2023-09-30T00:17:51.963382+0000","last_unstale":"2023-09-30T00:17:51.976127+0000","last_undegraded":"2023-09-30T00:17:51.976127+0000","last_fullsized":"2023-09-30T00:17:51.976127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:35:22.077360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972764+0000","last_change":"2023-09-30T00:17:51.963058+0000","last_active":"2023-09-30T00:17:51.972764+0000","last_peered":"2023-09-30T00:17:51.972764+0000","last_clean":"2023-09-30T00:17:51.972764+0000","last_became_active":"2023-09-30T00:17:51.959450+0000","last_became_peered":"2023-09-30T00:17:51.959450+0000","last_unstale":"2023-09-30T00:17:51.972764+0000","last_undegraded":"2023-09-30T00:17:51.972764+0000","last_fullsized":"2023-09-30T00:17:51.972764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:41:36.585190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975442+0000","last_change":"2023-09-30T00:17:51.967581+0000","last_active":"2023-09-30T00:17:51.975442+0000","last_peered":"2023-09-30T00:17:51.975442+0000","last_clean":"2023-09-30T00:17:51.975442+0000","last_became_active":"2023-09-30T00:17:51.961348+0000","last_became_peered":"2023-09-30T00:17:51.961348+0000","last_unstale":"2023-09-30T00:17:51.975442+0000","last_undegraded":"2023-09-30T00:17:51.975442+0000","last_fullsized":"2023-09-30T00:17:51.975442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:11:09.140469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975871+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.975871+0000","last_peered":"2023-09-30T00:17:51.975871+0000","last_clean":"2023-09-30T00:17:51.975871+0000","last_became_active":"2023-09-30T00:17:51.963534+0000","last_became_peered":"2023-09-30T00:17:51.963534+0000","last_unstale":"2023-09-30T00:17:51.975871+0000","last_undegraded":"2023-09-30T00:17:51.975871+0000","last_fullsized":"2023-09-30T00:17:51.975871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:21:49.086269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.569843+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:31.844594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972810+0000","last_change":"2023-09-30T00:17:51.968803+0000","last_active":"2023-09-30T00:17:51.972810+0000","last_peered":"2023-09-30T00:17:51.972810+0000","last_clean":"2023-09-30T00:17:51.972810+0000","last_became_active":"2023-09-30T00:17:51.965833+0000","last_became_peered":"2023-09-30T00:17:51.965833+0000","last_unstale":"2023-09-30T00:17:51.972810+0000","last_undegraded":"2023-09-30T00:17:51.972810+0000","last_fullsized":"2023-09-30T00:17:51.972810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960606+0000","last_change":"2023-09-30T00:17:51.592497+0000","last_active":"2023-09-30T00:17:51.960606+0000","last_peered":"2023-09-30T00:17:51.960606+0000","last_clean":"2023-09-30T00:17:51.960606+0000","last_became_active":"2023-09-30T00:17:51.585128+0000","last_became_peered":"2023-09-30T00:17:51.585128+0000","last_unstale":"2023-09-30T00:17:51.960606+0000","last_undegraded":"2023-09-30T00:17:51.960606+0000","last_fullsized":"2023-09-30T00:17:51.960606+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:46.058014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977744+0000","last_change":"2023-09-30T00:17:51.974520+0000","last_active":"2023-09-30T00:17:51.977744+0000","last_peered":"2023-09-30T00:17:51.977744+0000","last_clean":"2023-09-30T00:17:51.977744+0000","last_became_active":"2023-09-30T00:17:51.974109+0000","last_became_peered":"2023-09-30T00:17:51.974109+0000","last_unstale":"2023-09-30T00:17:51.977744+0000","last_undegraded":"2023-09-30T00:17:51.977744+0000","last_fullsized":"2023-09-30T00:17:51.977744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:25:05.134336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941011+0000","last_change":"2023-09-30T00:17:51.587986+0000","last_active":"2023-09-30T00:17:51.941011+0000","last_peered":"2023-09-30T00:17:51.941011+0000","last_clean":"2023-09-30T00:17:51.941011+0000","last_became_active":"2023-09-30T00:17:51.587724+0000","last_became_peered":"2023-09-30T00:17:51.587724+0000","last_unstale":"2023-09-30T00:17:51.941011+0000","last_undegraded":"2023-09-30T00:17:51.941011+0000","last_fullsized":"2023-09-30T00:17:51.941011+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568369+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:56:13.933771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952700+0000","last_change":"2023-09-30T00:17:51.596511+0000","last_active":"2023-09-30T00:17:51.952700+0000","last_peered":"2023-09-30T00:17:51.952700+0000","last_clean":"2023-09-30T00:17:51.952700+0000","last_became_active":"2023-09-30T00:17:51.596034+0000","last_became_peered":"2023-09-30T00:17:51.596034+0000","last_unstale":"2023-09-30T00:17:51.952700+0000","last_undegraded":"2023-09-30T00:17:51.952700+0000","last_fullsized":"2023-09-30T00:17:51.952700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:44:29.988610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969658+0000","last_change":"2023-09-30T00:17:51.964766+0000","last_active":"2023-09-30T00:17:51.969658+0000","last_peered":"2023-09-30T00:17:51.969658+0000","last_clean":"2023-09-30T00:17:51.969658+0000","last_became_active":"2023-09-30T00:17:51.964496+0000","last_became_peered":"2023-09-30T00:17:51.964496+0000","last_unstale":"2023-09-30T00:17:51.969658+0000","last_undegraded":"2023-09-30T00:17:51.969658+0000","last_fullsized":"2023-09-30T00:17:51.969658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.942081+0000","last_change":"2023-09-30T00:17:51.577022+0000","last_active":"2023-09-30T00:17:51.942081+0000","last_peered":"2023-09-30T00:17:51.942081+0000","last_clean":"2023-09-30T00:17:51.942081+0000","last_became_active":"2023-09-30T00:17:51.576726+0000","last_became_peered":"2023-09-30T00:17:51.576726+0000","last_unstale":"2023-09-30T00:17:51.942081+0000","last_undegraded":"2023-09-30T00:17:51.942081+0000","last_fullsized":"2023-09-30T00:17:51.942081+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972739+0000","last_change":"2023-09-30T00:17:51.969564+0000","last_active":"2023-09-30T00:17:51.972739+0000","last_peered":"2023-09-30T00:17:51.972739+0000","last_clean":"2023-09-30T00:17:51.972739+0000","last_became_active":"2023-09-30T00:17:51.969240+0000","last_became_peered":"2023-09-30T00:17:51.969240+0000","last_unstale":"2023-09-30T00:17:51.972739+0000","last_undegraded":"2023-09-30T00:17:51.972739+0000","last_fullsized":"2023-09-30T00:17:51.972739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:42.701890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.589757+0000","last_change":"2023-09-30T00:17:51.588716+0000","last_active":"2023-09-30T00:17:51.589757+0000","last_peered":"2023-09-30T00:17:51.589757+0000","last_clean":"2023-09-30T00:17:51.589757+0000","last_became_active":"2023-09-30T00:17:51.588454+0000","last_became_peered":"2023-09-30T00:17:51.588454+0000","last_unstale":"2023-09-30T00:17:51.589757+0000","last_undegraded":"2023-09-30T00:17:51.589757+0000","last_fullsized":"2023-09-30T00:17:51.589757+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568152+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:48:09.502104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"18'3","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965653+0000","last_change":"2023-09-30T00:17:51.589988+0000","last_active":"2023-09-30T00:17:51.965653+0000","last_peered":"2023-09-30T00:17:51.965653+0000","last_clean":"2023-09-30T00:17:51.965653+0000","last_became_active":"2023-09-30T00:17:51.589729+0000","last_became_peered":"2023-09-30T00:17:51.589729+0000","last_unstale":"2023-09-30T00:17:51.965653+0000","last_undegraded":"2023-09-30T00:17:51.965653+0000","last_fullsized":"2023-09-30T00:17:51.965653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975239+0000","last_change":"2023-09-30T00:17:51.967245+0000","last_active":"2023-09-30T00:17:51.975239+0000","last_peered":"2023-09-30T00:17:51.975239+0000","last_clean":"2023-09-30T00:17:51.975239+0000","last_became_active":"2023-09-30T00:17:51.963755+0000","last_became_peered":"2023-09-30T00:17:51.963755+0000","last_unstale":"2023-09-30T00:17:51.975239+0000","last_undegraded":"2023-09-30T00:17:51.975239+0000","last_fullsized":"2023-09-30T00:17:51.975239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:57:31.735059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"18'6","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.983073+0000","last_change":"2023-09-30T00:17:51.592307+0000","last_active":"2023-09-30T00:17:51.983073+0000","last_peered":"2023-09-30T00:17:51.983073+0000","last_clean":"2023-09-30T00:17:51.983073+0000","last_became_active":"2023-09-30T00:17:51.591753+0000","last_became_peered":"2023-09-30T00:17:51.591753+0000","last_unstale":"2023-09-30T00:17:51.983073+0000","last_undegraded":"2023-09-30T00:17:51.983073+0000","last_fullsized":"2023-09-30T00:17:51.983073+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:14:46.193926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.967859+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.967859+0000","last_peered":"2023-09-30T00:17:51.967859+0000","last_clean":"2023-09-30T00:17:51.967859+0000","last_became_active":"2023-09-30T00:17:51.967118+0000","last_became_peered":"2023-09-30T00:17:51.967118+0000","last_unstale":"2023-09-30T00:17:51.967859+0000","last_undegraded":"2023-09-30T00:17:51.967859+0000","last_fullsized":"2023-09-30T00:17:51.967859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:50.694814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969722+0000","last_change":"2023-09-30T00:17:51.591516+0000","last_active":"2023-09-30T00:17:51.969722+0000","last_peered":"2023-09-30T00:17:51.969722+0000","last_clean":"2023-09-30T00:17:51.969722+0000","last_became_active":"2023-09-30T00:17:51.591206+0000","last_became_peered":"2023-09-30T00:17:51.591206+0000","last_unstale":"2023-09-30T00:17:51.969722+0000","last_undegraded":"2023-09-30T00:17:51.969722+0000","last_fullsized":"2023-09-30T00:17:51.969722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:22.807978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977754+0000","last_change":"2023-09-30T00:17:51.974628+0000","last_active":"2023-09-30T00:17:51.977754+0000","last_peered":"2023-09-30T00:17:51.977754+0000","last_clean":"2023-09-30T00:17:51.977754+0000","last_became_active":"2023-09-30T00:17:51.974276+0000","last_became_peered":"2023-09-30T00:17:51.974276+0000","last_unstale":"2023-09-30T00:17:51.977754+0000","last_undegraded":"2023-09-30T00:17:51.977754+0000","last_fullsized":"2023-09-30T00:17:51.977754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.590891+0000","last_change":"2023-09-30T00:17:51.589138+0000","last_active":"2023-09-30T00:17:51.590891+0000","last_peered":"2023-09-30T00:17:51.590891+0000","last_clean":"2023-09-30T00:17:51.590891+0000","last_became_active":"2023-09-30T00:17:51.588883+0000","last_became_peered":"2023-09-30T00:17:51.588883+0000","last_unstale":"2023-09-30T00:17:51.590891+0000","last_undegraded":"2023-09-30T00:17:51.590891+0000","last_fullsized":"2023-09-30T00:17:51.590891+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977625+0000","last_change":"2023-09-30T00:17:51.972690+0000","last_active":"2023-09-30T00:17:51.977625+0000","last_peered":"2023-09-30T00:17:51.977625+0000","last_clean":"2023-09-30T00:17:51.977625+0000","last_became_active":"2023-09-30T00:17:51.969872+0000","last_became_peered":"2023-09-30T00:17:51.969872+0000","last_unstale":"2023-09-30T00:17:51.977625+0000","last_undegraded":"2023-09-30T00:17:51.977625+0000","last_fullsized":"2023-09-30T00:17:51.977625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952325+0000","last_change":"2023-09-30T00:17:51.597410+0000","last_active":"2023-09-30T00:17:51.952325+0000","last_peered":"2023-09-30T00:17:51.952325+0000","last_clean":"2023-09-30T00:17:51.952325+0000","last_became_active":"2023-09-30T00:17:51.597168+0000","last_became_peered":"2023-09-30T00:17:51.597168+0000","last_unstale":"2023-09-30T00:17:51.952325+0000","last_undegraded":"2023-09-30T00:17:51.952325+0000","last_fullsized":"2023-09-30T00:17:51.952325+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:44:45.339994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973574+0000","last_change":"2023-09-30T00:17:51.969752+0000","last_active":"2023-09-30T00:17:51.973574+0000","last_peered":"2023-09-30T00:17:51.973574+0000","last_clean":"2023-09-30T00:17:51.973574+0000","last_became_active":"2023-09-30T00:17:51.968993+0000","last_became_peered":"2023-09-30T00:17:51.968993+0000","last_unstale":"2023-09-30T00:17:51.973574+0000","last_undegraded":"2023-09-30T00:17:51.973574+0000","last_fullsized":"2023-09-30T00:17:51.973574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"18'8","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974927+0000","last_change":"2023-09-30T00:17:51.592339+0000","last_active":"2023-09-30T00:17:51.974927+0000","last_peered":"2023-09-30T00:17:51.974927+0000","last_clean":"2023-09-30T00:17:51.974927+0000","last_became_active":"2023-09-30T00:17:51.592134+0000","last_became_peered":"2023-09-30T00:17:51.592134+0000","last_unstale":"2023-09-30T00:17:51.974927+0000","last_undegraded":"2023-09-30T00:17:51.974927+0000","last_fullsized":"2023-09-30T00:17:51.974927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.569711+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:42:10.868448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"18'1","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.944741+0000","last_change":"2023-09-30T00:17:51.577740+0000","last_active":"2023-09-30T00:17:51.944741+0000","last_peered":"2023-09-30T00:17:51.944741+0000","last_clean":"2023-09-30T00:17:51.944741+0000","last_became_active":"2023-09-30T00:17:51.577449+0000","last_became_peered":"2023-09-30T00:17:51.577449+0000","last_unstale":"2023-09-30T00:17:51.944741+0000","last_undegraded":"2023-09-30T00:17:51.944741+0000","last_fullsized":"2023-09-30T00:17:51.944741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977756+0000","last_change":"2023-09-30T00:17:51.974425+0000","last_active":"2023-09-30T00:17:51.977756+0000","last_peered":"2023-09-30T00:17:51.977756+0000","last_clean":"2023-09-30T00:17:51.977756+0000","last_became_active":"2023-09-30T00:17:51.974143+0000","last_became_peered":"2023-09-30T00:17:51.974143+0000","last_unstale":"2023-09-30T00:17:51.977756+0000","last_undegraded":"2023-09-30T00:17:51.977756+0000","last_fullsized":"2023-09-30T00:17:51.977756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:04:54.968664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"18'2","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968091+0000","last_change":"2023-09-30T00:17:51.578710+0000","last_active":"2023-09-30T00:17:51.968091+0000","last_peered":"2023-09-30T00:17:51.968091+0000","last_clean":"2023-09-30T00:17:51.968091+0000","last_became_active":"2023-09-30T00:17:51.578450+0000","last_became_peered":"2023-09-30T00:17:51.578450+0000","last_unstale":"2023-09-30T00:17:51.968091+0000","last_undegraded":"2023-09-30T00:17:51.968091+0000","last_fullsized":"2023-09-30T00:17:51.968091+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.958310+0000","last_change":"2023-09-30T00:17:51.951367+0000","last_active":"2023-09-30T00:17:51.958310+0000","last_peered":"2023-09-30T00:17:51.958310+0000","last_clean":"2023-09-30T00:17:51.958310+0000","last_became_active":"2023-09-30T00:17:51.951031+0000","last_became_peered":"2023-09-30T00:17:51.951031+0000","last_unstale":"2023-09-30T00:17:51.958310+0000","last_undegraded":"2023-09-30T00:17:51.958310+0000","last_fullsized":"2023-09-30T00:17:51.958310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976462+0000","last_change":"2023-09-30T00:17:51.594234+0000","last_active":"2023-09-30T00:17:51.976462+0000","last_peered":"2023-09-30T00:17:51.976462+0000","last_clean":"2023-09-30T00:17:51.976462+0000","last_became_active":"2023-09-30T00:17:51.594049+0000","last_became_peered":"2023-09-30T00:17:51.594049+0000","last_unstale":"2023-09-30T00:17:51.976462+0000","last_undegraded":"2023-09-30T00:17:51.976462+0000","last_fullsized":"2023-09-30T00:17:51.976462+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:53:48.206055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965298+0000","last_change":"2023-09-30T00:17:51.965298+0000","last_active":"2023-09-30T00:17:51.965298+0000","last_peered":"2023-09-30T00:17:51.965298+0000","last_clean":"2023-09-30T00:17:51.965298+0000","last_became_active":"2023-09-30T00:17:51.964965+0000","last_became_peered":"2023-09-30T00:17:51.964965+0000","last_unstale":"2023-09-30T00:17:51.965298+0000","last_undegraded":"2023-09-30T00:17:51.965298+0000","last_fullsized":"2023-09-30T00:17:51.965298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:33:20.583162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.955628+0000","last_change":"2023-09-30T00:17:51.592485+0000","last_active":"2023-09-30T00:17:51.955628+0000","last_peered":"2023-09-30T00:17:51.955628+0000","last_clean":"2023-09-30T00:17:51.955628+0000","last_became_active":"2023-09-30T00:17:51.592054+0000","last_became_peered":"2023-09-30T00:17:51.592054+0000","last_unstale":"2023-09-30T00:17:51.955628+0000","last_undegraded":"2023-09-30T00:17:51.955628+0000","last_fullsized":"2023-09-30T00:17:51.955628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:46:52.872573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952057+0000","last_change":"2023-09-30T00:17:51.952057+0000","last_active":"2023-09-30T00:17:51.952057+0000","last_peered":"2023-09-30T00:17:51.952057+0000","last_clean":"2023-09-30T00:17:51.952057+0000","last_became_active":"2023-09-30T00:17:51.951537+0000","last_became_peered":"2023-09-30T00:17:51.951537+0000","last_unstale":"2023-09-30T00:17:51.952057+0000","last_undegraded":"2023-09-30T00:17:51.952057+0000","last_fullsized":"2023-09-30T00:17:51.952057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:20:25.199612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.578071+0000","last_change":"2023-09-30T00:17:51.578071+0000","last_active":"2023-09-30T00:17:51.578071+0000","last_peered":"2023-09-30T00:17:51.578071+0000","last_clean":"2023-09-30T00:17:51.578071+0000","last_became_active":"2023-09-30T00:17:51.577808+0000","last_became_peered":"2023-09-30T00:17:51.577808+0000","last_unstale":"2023-09-30T00:17:51.578071+0000","last_undegraded":"2023-09-30T00:17:51.578071+0000","last_fullsized":"2023-09-30T00:17:51.578071+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:20.626316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976183+0000","last_change":"2023-09-30T00:17:51.970822+0000","last_active":"2023-09-30T00:17:51.976183+0000","last_peered":"2023-09-30T00:17:51.976183+0000","last_clean":"2023-09-30T00:17:51.976183+0000","last_became_active":"2023-09-30T00:17:51.970520+0000","last_became_peered":"2023-09-30T00:17:51.970520+0000","last_unstale":"2023-09-30T00:17:51.976183+0000","last_undegraded":"2023-09-30T00:17:51.976183+0000","last_fullsized":"2023-09-30T00:17:51.976183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:07.347359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952250+0000","last_change":"2023-09-30T00:17:51.595723+0000","last_active":"2023-09-30T00:17:51.952250+0000","last_peered":"2023-09-30T00:17:51.952250+0000","last_clean":"2023-09-30T00:17:51.952250+0000","last_became_active":"2023-09-30T00:17:51.595473+0000","last_became_peered":"2023-09-30T00:17:51.595473+0000","last_unstale":"2023-09-30T00:17:51.952250+0000","last_undegraded":"2023-09-30T00:17:51.952250+0000","last_fullsized":"2023-09-30T00:17:51.952250+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:06.343890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980343+0000","last_change":"2023-09-30T00:17:51.977010+0000","last_active":"2023-09-30T00:17:51.980343+0000","last_peered":"2023-09-30T00:17:51.980343+0000","last_clean":"2023-09-30T00:17:51.980343+0000","last_became_active":"2023-09-30T00:17:51.976744+0000","last_became_peered":"2023-09-30T00:17:51.976744+0000","last_unstale":"2023-09-30T00:17:51.980343+0000","last_undegraded":"2023-09-30T00:17:51.980343+0000","last_fullsized":"2023-09-30T00:17:51.980343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:02:14.797729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952109+0000","last_change":"2023-09-30T00:17:51.595305+0000","last_active":"2023-09-30T00:17:51.952109+0000","last_peered":"2023-09-30T00:17:51.952109+0000","last_clean":"2023-09-30T00:17:51.952109+0000","last_became_active":"2023-09-30T00:17:51.595042+0000","last_became_peered":"2023-09-30T00:17:51.595042+0000","last_unstale":"2023-09-30T00:17:51.952109+0000","last_undegraded":"2023-09-30T00:17:51.952109+0000","last_fullsized":"2023-09-30T00:17:51.952109+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:54.681232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965816+0000","last_change":"2023-09-30T00:17:51.965816+0000","last_active":"2023-09-30T00:17:51.965816+0000","last_peered":"2023-09-30T00:17:51.965816+0000","last_clean":"2023-09-30T00:17:51.965816+0000","last_became_active":"2023-09-30T00:17:51.965549+0000","last_became_peered":"2023-09-30T00:17:51.965549+0000","last_unstale":"2023-09-30T00:17:51.965816+0000","last_undegraded":"2023-09-30T00:17:51.965816+0000","last_fullsized":"2023-09-30T00:17:51.965816+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:30.526216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.953542+0000","last_change":"2023-09-30T00:17:51.596289+0000","last_active":"2023-09-30T00:17:51.953542+0000","last_peered":"2023-09-30T00:17:51.953542+0000","last_clean":"2023-09-30T00:17:51.953542+0000","last_became_active":"2023-09-30T00:17:51.595689+0000","last_became_peered":"2023-09-30T00:17:51.595689+0000","last_unstale":"2023-09-30T00:17:51.953542+0000","last_undegraded":"2023-09-30T00:17:51.953542+0000","last_fullsized":"2023-09-30T00:17:51.953542+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:46:55.016362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975889+0000","last_change":"2023-09-30T00:17:51.970420+0000","last_active":"2023-09-30T00:17:51.975889+0000","last_peered":"2023-09-30T00:17:51.975889+0000","last_clean":"2023-09-30T00:17:51.975889+0000","last_became_active":"2023-09-30T00:17:51.963999+0000","last_became_peered":"2023-09-30T00:17:51.963999+0000","last_unstale":"2023-09-30T00:17:51.975889+0000","last_undegraded":"2023-09-30T00:17:51.975889+0000","last_fullsized":"2023-09-30T00:17:51.975889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962989+0000","last_change":"2023-09-30T00:17:51.590485+0000","last_active":"2023-09-30T00:17:51.962989+0000","last_peered":"2023-09-30T00:17:51.962989+0000","last_clean":"2023-09-30T00:17:51.962989+0000","last_became_active":"2023-09-30T00:17:51.590210+0000","last_became_peered":"2023-09-30T00:17:51.590210+0000","last_unstale":"2023-09-30T00:17:51.962989+0000","last_undegraded":"2023-09-30T00:17:51.962989+0000","last_fullsized":"2023-09-30T00:17:51.962989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:32.708671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.582012+0000","last_change":"2023-09-30T00:17:45.196688+0000","last_active":"2023-09-30T00:17:51.582012+0000","last_peered":"2023-09-30T00:17:51.582012+0000","last_clean":"2023-09-30T00:17:51.582012+0000","last_became_active":"2023-09-30T00:17:45.196361+0000","last_became_peered":"2023-09-30T00:17:45.196361+0000","last_unstale":"2023-09-30T00:17:51.582012+0000","last_undegraded":"2023-09-30T00:17:51.582012+0000","last_fullsized":"2023-09-30T00:17:51.582012+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977735+0000","last_change":"2023-09-30T00:17:51.974386+0000","last_active":"2023-09-30T00:17:51.977735+0000","last_peered":"2023-09-30T00:17:51.977735+0000","last_clean":"2023-09-30T00:17:51.977735+0000","last_became_active":"2023-09-30T00:17:51.973776+0000","last_became_peered":"2023-09-30T00:17:51.973776+0000","last_unstale":"2023-09-30T00:17:51.977735+0000","last_undegraded":"2023-09-30T00:17:51.977735+0000","last_fullsized":"2023-09-30T00:17:51.977735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:25:21.282436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:17:50.566871+0000","last_change":"2023-09-30T00:17:45.203067+0000","last_active":"2023-09-30T00:17:50.566871+0000","last_peered":"2023-09-30T00:17:50.566871+0000","last_clean":"2023-09-30T00:17:50.566871+0000","last_became_active":"2023-09-30T00:17:45.202823+0000","last_became_peered":"2023-09-30T00:17:45.202823+0000","last_unstale":"2023-09-30T00:17:50.566871+0000","last_undegraded":"2023-09-30T00:17:50.566871+0000","last_fullsized":"2023-09-30T00:17:50.566871+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:26:01.195302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943027+0000","last_change":"2023-09-30T00:17:51.592843+0000","last_active":"2023-09-30T00:17:51.943027+0000","last_peered":"2023-09-30T00:17:51.943027+0000","last_clean":"2023-09-30T00:17:51.943027+0000","last_became_active":"2023-09-30T00:17:51.592601+0000","last_became_peered":"2023-09-30T00:17:51.592601+0000","last_unstale":"2023-09-30T00:17:51.943027+0000","last_undegraded":"2023-09-30T00:17:51.943027+0000","last_fullsized":"2023-09-30T00:17:51.943027+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965286+0000","last_change":"2023-09-30T00:17:51.965286+0000","last_active":"2023-09-30T00:17:51.965286+0000","last_peered":"2023-09-30T00:17:51.965286+0000","last_clean":"2023-09-30T00:17:51.965286+0000","last_became_active":"2023-09-30T00:17:51.964869+0000","last_became_peered":"2023-09-30T00:17:51.964869+0000","last_unstale":"2023-09-30T00:17:51.965286+0000","last_undegraded":"2023-09-30T00:17:51.965286+0000","last_fullsized":"2023-09-30T00:17:51.965286+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:46.192135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:17:50.566873+0000","last_change":"2023-09-30T00:17:45.195737+0000","last_active":"2023-09-30T00:17:50.566873+0000","last_peered":"2023-09-30T00:17:50.566873+0000","last_clean":"2023-09-30T00:17:50.566873+0000","last_became_active":"2023-09-30T00:17:45.195464+0000","last_became_peered":"2023-09-30T00:17:45.195464+0000","last_unstale":"2023-09-30T00:17:50.566873+0000","last_undegraded":"2023-09-30T00:17:50.566873+0000","last_fullsized":"2023-09-30T00:17:50.566873+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:24:48.730978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952759+0000","last_change":"2023-09-30T00:17:47.199132+0000","last_active":"2023-09-30T00:17:51.952759+0000","last_peered":"2023-09-30T00:17:51.952759+0000","last_clean":"2023-09-30T00:17:51.952759+0000","last_became_active":"2023-09-30T00:17:47.198255+0000","last_became_peered":"2023-09-30T00:17:47.198255+0000","last_unstale":"2023-09-30T00:17:51.952759+0000","last_undegraded":"2023-09-30T00:17:51.952759+0000","last_fullsized":"2023-09-30T00:17:51.952759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_clean_scrub_stamp":"2023-09-30T00:17:46.189460+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:09.834201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.575046+0000","last_change":"2023-09-30T00:17:51.575046+0000","last_active":"2023-09-30T00:17:51.575046+0000","last_peered":"2023-09-30T00:17:51.575046+0000","last_clean":"2023-09-30T00:17:51.575046+0000","last_became_active":"2023-09-30T00:17:51.574815+0000","last_became_peered":"2023-09-30T00:17:51.574815+0000","last_unstale":"2023-09-30T00:17:51.575046+0000","last_undegraded":"2023-09-30T00:17:51.575046+0000","last_fullsized":"2023-09-30T00:17:51.575046+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:43:57.892326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973520+0000","last_change":"2023-09-30T00:17:51.969276+0000","last_active":"2023-09-30T00:17:51.973520+0000","last_peered":"2023-09-30T00:17:51.973520+0000","last_clean":"2023-09-30T00:17:51.973520+0000","last_became_active":"2023-09-30T00:17:51.966513+0000","last_became_peered":"2023-09-30T00:17:51.966513+0000","last_unstale":"2023-09-30T00:17:51.973520+0000","last_undegraded":"2023-09-30T00:17:51.973520+0000","last_fullsized":"2023-09-30T00:17:51.973520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.579721+0000","last_change":"2023-09-30T00:17:45.197230+0000","last_active":"2023-09-30T00:17:51.579721+0000","last_peered":"2023-09-30T00:17:51.579721+0000","last_clean":"2023-09-30T00:17:51.579721+0000","last_became_active":"2023-09-30T00:17:45.196528+0000","last_became_peered":"2023-09-30T00:17:45.196528+0000","last_unstale":"2023-09-30T00:17:51.579721+0000","last_undegraded":"2023-09-30T00:17:51.579721+0000","last_fullsized":"2023-09-30T00:17:51.579721+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.575520+0000","last_change":"2023-09-30T00:17:51.575520+0000","last_active":"2023-09-30T00:17:51.575520+0000","last_peered":"2023-09-30T00:17:51.575520+0000","last_clean":"2023-09-30T00:17:51.575520+0000","last_became_active":"2023-09-30T00:17:51.575306+0000","last_became_peered":"2023-09-30T00:17:51.575306+0000","last_unstale":"2023-09-30T00:17:51.575520+0000","last_undegraded":"2023-09-30T00:17:51.575520+0000","last_fullsized":"2023-09-30T00:17:51.575520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:31:32.127076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568586+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:37:36.608579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.940896+0000","last_change":"2023-09-30T00:17:45.196542+0000","last_active":"2023-09-30T00:17:51.940896+0000","last_peered":"2023-09-30T00:17:51.940896+0000","last_clean":"2023-09-30T00:17:51.940896+0000","last_became_active":"2023-09-30T00:17:45.196123+0000","last_became_peered":"2023-09-30T00:17:45.196123+0000","last_unstale":"2023-09-30T00:17:51.940896+0000","last_undegraded":"2023-09-30T00:17:51.940896+0000","last_fullsized":"2023-09-30T00:17:51.940896+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952371+0000","last_change":"2023-09-30T00:17:51.591518+0000","last_active":"2023-09-30T00:17:51.952371+0000","last_peered":"2023-09-30T00:17:51.952371+0000","last_clean":"2023-09-30T00:17:51.952371+0000","last_became_active":"2023-09-30T00:17:51.591253+0000","last_became_peered":"2023-09-30T00:17:51.591253+0000","last_unstale":"2023-09-30T00:17:51.952371+0000","last_undegraded":"2023-09-30T00:17:51.952371+0000","last_fullsized":"2023-09-30T00:17:51.952371+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:52:46.330854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980323+0000","last_change":"2023-09-30T00:17:51.976800+0000","last_active":"2023-09-30T00:17:51.980323+0000","last_peered":"2023-09-30T00:17:51.980323+0000","last_clean":"2023-09-30T00:17:51.980323+0000","last_became_active":"2023-09-30T00:17:51.976602+0000","last_became_peered":"2023-09-30T00:17:51.976602+0000","last_unstale":"2023-09-30T00:17:51.980323+0000","last_undegraded":"2023-09-30T00:17:51.980323+0000","last_fullsized":"2023-09-30T00:17:51.980323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:53:26.550299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952216+0000","last_change":"2023-09-30T00:17:45.207123+0000","last_active":"2023-09-30T00:17:51.952216+0000","last_peered":"2023-09-30T00:17:51.952216+0000","last_clean":"2023-09-30T00:17:51.952216+0000","last_became_active":"2023-09-30T00:17:45.206750+0000","last_became_peered":"2023-09-30T00:17:45.206750+0000","last_unstale":"2023-09-30T00:17:51.952216+0000","last_undegraded":"2023-09-30T00:17:51.952216+0000","last_fullsized":"2023-09-30T00:17:51.952216+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.587209+0000","last_change":"2023-09-30T00:17:51.585515+0000","last_active":"2023-09-30T00:17:51.587209+0000","last_peered":"2023-09-30T00:17:51.587209+0000","last_clean":"2023-09-30T00:17:51.587209+0000","last_became_active":"2023-09-30T00:17:51.585262+0000","last_became_peered":"2023-09-30T00:17:51.585262+0000","last_unstale":"2023-09-30T00:17:51.587209+0000","last_undegraded":"2023-09-30T00:17:51.587209+0000","last_fullsized":"2023-09-30T00:17:51.587209+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:04.914282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568596+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:59.954235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.954829+0000","last_change":"2023-09-30T00:17:45.206971+0000","last_active":"2023-09-30T00:17:51.954829+0000","last_peered":"2023-09-30T00:17:51.954829+0000","last_clean":"2023-09-30T00:17:51.954829+0000","last_became_active":"2023-09-30T00:17:45.206693+0000","last_became_peered":"2023-09-30T00:17:45.206693+0000","last_unstale":"2023-09-30T00:17:51.954829+0000","last_undegraded":"2023-09-30T00:17:51.954829+0000","last_fullsized":"2023-09-30T00:17:51.954829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.575506+0000","last_change":"2023-09-30T00:17:51.575506+0000","last_active":"2023-09-30T00:17:51.575506+0000","last_peered":"2023-09-30T00:17:51.575506+0000","last_clean":"2023-09-30T00:17:51.575506+0000","last_became_active":"2023-09-30T00:17:51.574806+0000","last_became_peered":"2023-09-30T00:17:51.574806+0000","last_unstale":"2023-09-30T00:17:51.575506+0000","last_undegraded":"2023-09-30T00:17:51.575506+0000","last_fullsized":"2023-09-30T00:17:51.575506+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:51.816991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975408+0000","last_change":"2023-09-30T00:17:51.967785+0000","last_active":"2023-09-30T00:17:51.975408+0000","last_peered":"2023-09-30T00:17:51.975408+0000","last_clean":"2023-09-30T00:17:51.975408+0000","last_became_active":"2023-09-30T00:17:51.967419+0000","last_became_peered":"2023-09-30T00:17:51.967419+0000","last_unstale":"2023-09-30T00:17:51.975408+0000","last_undegraded":"2023-09-30T00:17:51.975408+0000","last_fullsized":"2023-09-30T00:17:51.975408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941618+0000","last_change":"2023-09-30T00:17:45.203202+0000","last_active":"2023-09-30T00:17:51.941618+0000","last_peered":"2023-09-30T00:17:51.941618+0000","last_clean":"2023-09-30T00:17:51.941618+0000","last_became_active":"2023-09-30T00:17:45.202879+0000","last_became_peered":"2023-09-30T00:17:45.202879+0000","last_unstale":"2023-09-30T00:17:51.941618+0000","last_undegraded":"2023-09-30T00:17:51.941618+0000","last_fullsized":"2023-09-30T00:17:51.941618+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:50:42.379943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.585090+0000","last_change":"2023-09-30T00:17:51.580771+0000","last_active":"2023-09-30T00:17:51.585090+0000","last_peered":"2023-09-30T00:17:51.585090+0000","last_clean":"2023-09-30T00:17:51.585090+0000","last_became_active":"2023-09-30T00:17:51.580495+0000","last_became_peered":"2023-09-30T00:17:51.580495+0000","last_unstale":"2023-09-30T00:17:51.585090+0000","last_undegraded":"2023-09-30T00:17:51.585090+0000","last_fullsized":"2023-09-30T00:17:51.585090+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971832+0000","last_change":"2023-09-30T00:17:51.957745+0000","last_active":"2023-09-30T00:17:51.971832+0000","last_peered":"2023-09-30T00:17:51.971832+0000","last_clean":"2023-09-30T00:17:51.971832+0000","last_became_active":"2023-09-30T00:17:51.955331+0000","last_became_peered":"2023-09-30T00:17:51.955331+0000","last_unstale":"2023-09-30T00:17:51.971832+0000","last_undegraded":"2023-09-30T00:17:51.971832+0000","last_fullsized":"2023-09-30T00:17:51.971832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:45.393660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973877+0000","last_change":"2023-09-30T00:17:51.596021+0000","last_active":"2023-09-30T00:17:51.973877+0000","last_peered":"2023-09-30T00:17:51.973877+0000","last_clean":"2023-09-30T00:17:51.973877+0000","last_became_active":"2023-09-30T00:17:51.595745+0000","last_became_peered":"2023-09-30T00:17:51.595745+0000","last_unstale":"2023-09-30T00:17:51.973877+0000","last_undegraded":"2023-09-30T00:17:51.973877+0000","last_fullsized":"2023-09-30T00:17:51.973877+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:44.390191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.569794+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:06:14.351937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974567+0000","last_change":"2023-09-30T00:17:51.587276+0000","last_active":"2023-09-30T00:17:51.974567+0000","last_peered":"2023-09-30T00:17:51.974567+0000","last_clean":"2023-09-30T00:17:51.974567+0000","last_became_active":"2023-09-30T00:17:51.586984+0000","last_became_peered":"2023-09-30T00:17:51.586984+0000","last_unstale":"2023-09-30T00:17:51.974567+0000","last_undegraded":"2023-09-30T00:17:51.974567+0000","last_fullsized":"2023-09-30T00:17:51.974567+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.953787+0000","last_change":"2023-09-30T00:17:51.953787+0000","last_active":"2023-09-30T00:17:51.953787+0000","last_peered":"2023-09-30T00:17:51.953787+0000","last_clean":"2023-09-30T00:17:51.953787+0000","last_became_active":"2023-09-30T00:17:51.951805+0000","last_became_peered":"2023-09-30T00:17:51.951805+0000","last_unstale":"2023-09-30T00:17:51.953787+0000","last_undegraded":"2023-09-30T00:17:51.953787+0000","last_fullsized":"2023-09-30T00:17:51.953787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:57:04.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965610+0000","last_change":"2023-09-30T00:17:51.593225+0000","last_active":"2023-09-30T00:17:51.965610+0000","last_peered":"2023-09-30T00:17:51.965610+0000","last_clean":"2023-09-30T00:17:51.965610+0000","last_became_active":"2023-09-30T00:17:51.592913+0000","last_became_peered":"2023-09-30T00:17:51.592913+0000","last_unstale":"2023-09-30T00:17:51.965610+0000","last_undegraded":"2023-09-30T00:17:51.965610+0000","last_fullsized":"2023-09-30T00:17:51.965610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:37:56.776343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.955058+0000","last_change":"2023-09-30T00:17:51.955058+0000","last_active":"2023-09-30T00:17:51.955058+0000","last_peered":"2023-09-30T00:17:51.955058+0000","last_clean":"2023-09-30T00:17:51.955058+0000","last_became_active":"2023-09-30T00:17:51.954787+0000","last_became_peered":"2023-09-30T00:17:51.954787+0000","last_unstale":"2023-09-30T00:17:51.955058+0000","last_undegraded":"2023-09-30T00:17:51.955058+0000","last_fullsized":"2023-09-30T00:17:51.955058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:49:30.148218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.586229+0000","last_change":"2023-09-30T00:17:51.586229+0000","last_active":"2023-09-30T00:17:51.586229+0000","last_peered":"2023-09-30T00:17:51.586229+0000","last_clean":"2023-09-30T00:17:51.586229+0000","last_became_active":"2023-09-30T00:17:51.586027+0000","last_became_peered":"2023-09-30T00:17:51.586027+0000","last_unstale":"2023-09-30T00:17:51.586229+0000","last_undegraded":"2023-09-30T00:17:51.586229+0000","last_fullsized":"2023-09-30T00:17:51.586229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:48:13.817514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962963+0000","last_change":"2023-09-30T00:17:51.959628+0000","last_active":"2023-09-30T00:17:51.962963+0000","last_peered":"2023-09-30T00:17:51.962963+0000","last_clean":"2023-09-30T00:17:51.962963+0000","last_became_active":"2023-09-30T00:17:51.958974+0000","last_became_peered":"2023-09-30T00:17:51.958974+0000","last_unstale":"2023-09-30T00:17:51.962963+0000","last_undegraded":"2023-09-30T00:17:51.962963+0000","last_fullsized":"2023-09-30T00:17:51.962963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:28.604134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"18'4","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960480+0000","last_change":"2023-09-30T00:17:51.590387+0000","last_active":"2023-09-30T00:17:51.960480+0000","last_peered":"2023-09-30T00:17:51.960480+0000","last_clean":"2023-09-30T00:17:51.960480+0000","last_became_active":"2023-09-30T00:17:51.585479+0000","last_became_peered":"2023-09-30T00:17:51.585479+0000","last_unstale":"2023-09-30T00:17:51.960480+0000","last_undegraded":"2023-09-30T00:17:51.960480+0000","last_fullsized":"2023-09-30T00:17:51.960480+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.949418+0000","last_change":"2023-09-30T00:17:51.949418+0000","last_active":"2023-09-30T00:17:51.949418+0000","last_peered":"2023-09-30T00:17:51.949418+0000","last_clean":"2023-09-30T00:17:51.949418+0000","last_became_active":"2023-09-30T00:17:51.949101+0000","last_became_peered":"2023-09-30T00:17:51.949101+0000","last_unstale":"2023-09-30T00:17:51.949418+0000","last_undegraded":"2023-09-30T00:17:51.949418+0000","last_fullsized":"2023-09-30T00:17:51.949418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975274+0000","last_change":"2023-09-30T00:17:51.964155+0000","last_active":"2023-09-30T00:17:51.975274+0000","last_peered":"2023-09-30T00:17:51.975274+0000","last_clean":"2023-09-30T00:17:51.975274+0000","last_became_active":"2023-09-30T00:17:51.958334+0000","last_became_peered":"2023-09-30T00:17:51.958334+0000","last_unstale":"2023-09-30T00:17:51.975274+0000","last_undegraded":"2023-09-30T00:17:51.975274+0000","last_fullsized":"2023-09-30T00:17:51.975274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570057+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:31:59.960264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976104+0000","last_change":"2023-09-30T00:17:51.970791+0000","last_active":"2023-09-30T00:17:51.976104+0000","last_peered":"2023-09-30T00:17:51.976104+0000","last_clean":"2023-09-30T00:17:51.976104+0000","last_became_active":"2023-09-30T00:17:51.964142+0000","last_became_peered":"2023-09-30T00:17:51.964142+0000","last_unstale":"2023-09-30T00:17:51.976104+0000","last_undegraded":"2023-09-30T00:17:51.976104+0000","last_fullsized":"2023-09-30T00:17:51.976104+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568144+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:18:28.873560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570896+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:02:47.475511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672965,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9220,"kb_used_data":764,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362620,"statfs":{"total":96636764160,"available":96627322880,"internally_reserved":0,"allocated":782336,"data_stored":646622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]}]},{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8060,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363780,"statfs":{"total":96636764160,"available":96628510720,"internally_reserved":0,"allocated":184320,"data_stored":55478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8852,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94362988,"statfs":{"total":96636764160,"available":96627699712,"internally_reserved":0,"allocated":733184,"data_stored":634680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8052,"kb_used_data":172,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363788,"statfs":{"total":96636764160,"available":96628518912,"internally_reserved":0,"allocated":176128,"data_stored":55170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.859}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7779999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:17:56.352 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:17:56.496 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.494+0000 7f6b5882e700 1 Processor -- start 2023-09-30T00:17:56.496 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.494+0000 7f6b5882e700 1 -- start start 2023-09-30T00:17:56.497 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.494+0000 7f6b5882e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6b54067e30 0x7f6b540a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.497 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.494+0000 7f6b5882e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b540a9ea0 0x7f6b5413fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.497 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.494+0000 7f6b5882e700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b54068200 con 0x7f6b540a94d0 2023-09-30T00:17:56.497 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.494+0000 7f6b5882e700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b540a22b0 con 0x7f6b54067e30 2023-09-30T00:17:56.497 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.494+0000 7f6b5882e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b540a24a0 con 0x7f6b540a9ea0 2023-09-30T00:17:56.497 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.494+0000 7f6b49d9b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b540a9ea0 0x7f6b5413fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.497 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.494+0000 7f6b49d9b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b540a9ea0 0x7f6b5413fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39040/0 (socket says 172.21.15.94:39040) 2023-09-30T00:17:56.497 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.494+0000 7f6b49d9b700 1 -- 172.21.15.94:0/599200064 learned_addr learned my addr 172.21.15.94:0/599200064 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:56.498 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.494+0000 7f6b49d9b700 1 -- 172.21.15.94:0/599200064 >> v2:172.21.15.121:3300/0 conn(0x7f6b54067e30 msgr2=0x7f6b540a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:56.498 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.494+0000 7f6b5259c700 1 --2- 172.21.15.94:0/599200064 >> v2:172.21.15.121:3300/0 conn(0x7f6b54067e30 0x7f6b540a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.498 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b49d9b700 1 --2- 172.21.15.94:0/599200064 >> v2:172.21.15.121:3300/0 conn(0x7f6b54067e30 0x7f6b540a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.498 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b49d9b700 1 -- 172.21.15.94:0/599200064 >> v1:172.21.15.121:6789/0 conn(0x7f6b540a94d0 legacy=0x7f6b541402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:56.498 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b49d9b700 1 -- 172.21.15.94:0/599200064 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b540a2740 con 0x7f6b540a9ea0 2023-09-30T00:17:56.498 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b5259c700 1 --2- 172.21.15.94:0/599200064 >> v2:172.21.15.121:3300/0 conn(0x7f6b54067e30 0x7f6b540a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:17:56.498 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b49d9b700 1 --2- 172.21.15.94:0/599200064 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b540a9ea0 0x7f6b5413fd60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0099b0 tx=0x7f6b3c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61b3c41cbb52f233 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:56.498 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b51d9b700 1 -- 172.21.15.94:0/599200064 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c00b820 con 0x7f6b540a9ea0 2023-09-30T00:17:56.498 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b51d9b700 1 -- 172.21.15.94:0/599200064 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c00b980 con 0x7f6b540a9ea0 2023-09-30T00:17:56.498 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b51d9b700 1 -- 172.21.15.94:0/599200064 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c00bd20 con 0x7f6b540a9ea0 2023-09-30T00:17:56.499 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b5882e700 1 -- 172.21.15.94:0/599200064 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b540a9ea0 msgr2=0x7f6b5413fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.499 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b5882e700 1 --2- 172.21.15.94:0/599200064 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b540a9ea0 0x7f6b5413fd60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0099b0 tx=0x7f6b3c009250 comp rx=0 tx=0).stop 2023-09-30T00:17:56.499 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b5882e700 1 -- 172.21.15.94:0/599200064 shutdown_connections 2023-09-30T00:17:56.499 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b5882e700 1 --2- 172.21.15.94:0/599200064 >> v2:172.21.15.121:3300/0 conn(0x7f6b54067e30 0x7f6b540a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.499 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b5882e700 1 --2- 172.21.15.94:0/599200064 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b540a9ea0 0x7f6b5413fd60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.499 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b5882e700 1 -- 172.21.15.94:0/599200064 >> 172.21.15.94:0/599200064 conn(0x7f6b540702f0 msgr2=0x7f6b540706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:56.499 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b5882e700 1 -- 172.21.15.94:0/599200064 shutdown_connections 2023-09-30T00:17:56.499 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.495+0000 7f6b5882e700 1 -- 172.21.15.94:0/599200064 wait complete. 2023-09-30T00:17:56.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.496+0000 7f6b5882e700 1 Processor -- start 2023-09-30T00:17:56.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.496+0000 7f6b5882e700 1 -- start start 2023-09-30T00:17:56.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5882e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b54067e30 0x7f6b54143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5882e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b540a94d0 0x7f6b541426d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5882e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6b541448f0 0x7f6b54150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5882e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b541508e0 con 0x7f6b541448f0 2023-09-30T00:17:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5882e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b54150b40 con 0x7f6b540a94d0 2023-09-30T00:17:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5882e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b54143260 con 0x7f6b54067e30 2023-09-30T00:17:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5259c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b54067e30 0x7f6b54143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5259c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b54067e30 0x7f6b54143cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39048/0 (socket says 172.21.15.94:39048) 2023-09-30T00:17:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5259c700 1 -- 172.21.15.94:0/4046659352 learned_addr learned my addr 172.21.15.94:0/4046659352 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b52d9d700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6b541448f0 0x7f6b54150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b49d9b700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b540a94d0 0x7f6b541426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5259c700 1 -- 172.21.15.94:0/4046659352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b540a94d0 msgr2=0x7f6b541426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5259c700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b540a94d0 0x7f6b541426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5259c700 1 -- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6b541448f0 msgr2=0x7f6b54150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5259c700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6b541448f0 0x7f6b54150e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5259c700 1 -- 172.21.15.94:0/4046659352 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c0094f0 con 0x7f6b54067e30 2023-09-30T00:17:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.497+0000 7f6b5259c700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b54067e30 0x7f6b54143cb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400c830 tx=0x7f6b44008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.498+0000 7f6b4affd700 1 -- 172.21.15.94:0/4046659352 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b44011600 con 0x7f6b54067e30 2023-09-30T00:17:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.498+0000 7f6b4affd700 1 -- 172.21.15.94:0/4046659352 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b44011760 con 0x7f6b54067e30 2023-09-30T00:17:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.498+0000 7f6b5882e700 1 -- 172.21.15.94:0/4046659352 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b541443e0 con 0x7f6b54067e30 2023-09-30T00:17:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.498+0000 7f6b4affd700 1 -- 172.21.15.94:0/4046659352 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b44011b20 con 0x7f6b54067e30 2023-09-30T00:17:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.498+0000 7f6b5882e700 1 -- 172.21.15.94:0/4046659352 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b540a61c0 con 0x7f6b54067e30 2023-09-30T00:17:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.499+0000 7f6b5882e700 1 -- 172.21.15.94:0/4046659352 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b540a6410 con 0x7f6b54067e30 2023-09-30T00:17:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.500+0000 7f6b4affd700 1 -- 172.21.15.94:0/4046659352 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6b44024480 con 0x7f6b54067e30 2023-09-30T00:17:56.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.500+0000 7f6b4affd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:56.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.500+0000 7f6b4affd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:56.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.500+0000 7f6b4affd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:56.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.500+0000 7f6b4affd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:56.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.500+0000 7f6b4affd700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6b34098780 0x7f6b3409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.500+0000 7f6b49d9b700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6b34098780 0x7f6b3409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.500+0000 7f6b4affd700 1 -- 172.21.15.94:0/4046659352 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f6b440c06b0 con 0x7f6b54067e30 2023-09-30T00:17:56.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.500+0000 7f6b49d9b700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6b34098780 0x7f6b3409ac30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0099b0 tx=0x7f6b3c016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:56.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.503+0000 7f6b4affd700 1 -- 172.21.15.94:0/4046659352 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6b4408ddf0 con 0x7f6b54067e30 2023-09-30T00:17:56.683 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.682+0000 7f6b5882e700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-30T00:17:56.683 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.682+0000 7f6b5882e700 1 -- 172.21.15.94:0/4046659352 --> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b540a6410 con 0x7f6b34098780 2023-09-30T00:17:56.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.689+0000 7f6b4affd700 1 -- 172.21.15.94:0/4046659352 <== mgr.4106 v2:172.21.15.94:6808/84242 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+268834 (secure 0 0 0) 0x7f6b540a6410 con 0x7f6b34098780 2023-09-30T00:17:56.690 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.689+0000 7f6b4affd700 20 mgrc handle_command_reply tid 0 r 0 2023-09-30T00:17:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:56.696 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 10 mgrc shutdown 2023-09-30T00:17:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 1 -- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6b34098780 msgr2=0x7f6b3409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6b34098780 0x7f6b3409ac30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0099b0 tx=0x7f6b3c016000 comp rx=0 tx=0).stop 2023-09-30T00:17:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 1 -- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b54067e30 msgr2=0x7f6b54143cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b54067e30 0x7f6b54143cb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400c830 tx=0x7f6b44008040 comp rx=0 tx=0).stop 2023-09-30T00:17:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 1 -- 172.21.15.94:0/4046659352 shutdown_connections 2023-09-30T00:17:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6b34098780 0x7f6b3409ac30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6b54067e30 0x7f6b54143cb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6b541448f0 0x7f6b54150e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 1 --2- 172.21.15.94:0/4046659352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b540a94d0 0x7f6b541426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 1 -- 172.21.15.94:0/4046659352 >> 172.21.15.94:0/4046659352 conn(0x7f6b540702f0 msgr2=0x7f6b540b7580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:56.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 1 -- 172.21.15.94:0/4046659352 shutdown_connections 2023-09-30T00:17:56.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.693+0000 7f6b5882e700 1 -- 172.21.15.94:0/4046659352 wait complete. 2023-09-30T00:17:56.698 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2023-09-30T00:17:56.705 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-09-30T00:17:55.851278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":202,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":34184,"kb_used_data":1832,"kb_used_omap":0,"kb_used_meta":32320,"kb_avail":377453176,"statfs":{"total":386547056640,"available":386512052224,"internally_reserved":0,"allocated":1875968,"data_stored":1391950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33095680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.286619"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960193+0000","last_change":"2023-09-30T00:17:51.960193+0000","last_active":"2023-09-30T00:17:51.960193+0000","last_peered":"2023-09-30T00:17:51.960193+0000","last_clean":"2023-09-30T00:17:51.960193+0000","last_became_active":"2023-09-30T00:17:51.959888+0000","last_became_peered":"2023-09-30T00:17:51.959888+0000","last_unstale":"2023-09-30T00:17:51.960193+0000","last_undegraded":"2023-09-30T00:17:51.960193+0000","last_fullsized":"2023-09-30T00:17:51.960193+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:40:42.458928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968577+0000","last_change":"2023-09-30T00:17:51.968577+0000","last_active":"2023-09-30T00:17:51.968577+0000","last_peered":"2023-09-30T00:17:51.968577+0000","last_clean":"2023-09-30T00:17:51.968577+0000","last_became_active":"2023-09-30T00:17:51.968275+0000","last_became_peered":"2023-09-30T00:17:51.968275+0000","last_unstale":"2023-09-30T00:17:51.968577+0000","last_undegraded":"2023-09-30T00:17:51.968577+0000","last_fullsized":"2023-09-30T00:17:51.968577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:19:24.974156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977703+0000","last_change":"2023-09-30T00:17:51.974359+0000","last_active":"2023-09-30T00:17:51.977703+0000","last_peered":"2023-09-30T00:17:51.977703+0000","last_clean":"2023-09-30T00:17:51.977703+0000","last_became_active":"2023-09-30T00:17:51.973789+0000","last_became_peered":"2023-09-30T00:17:51.973789+0000","last_unstale":"2023-09-30T00:17:51.977703+0000","last_undegraded":"2023-09-30T00:17:51.977703+0000","last_fullsized":"2023-09-30T00:17:51.977703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:27.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971859+0000","last_change":"2023-09-30T00:17:51.958127+0000","last_active":"2023-09-30T00:17:51.971859+0000","last_peered":"2023-09-30T00:17:51.971859+0000","last_clean":"2023-09-30T00:17:51.971859+0000","last_became_active":"2023-09-30T00:17:51.957850+0000","last_became_peered":"2023-09-30T00:17:51.957850+0000","last_unstale":"2023-09-30T00:17:51.971859+0000","last_undegraded":"2023-09-30T00:17:51.971859+0000","last_fullsized":"2023-09-30T00:17:51.971859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:48:36.950659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973474+0000","last_change":"2023-09-30T00:17:51.966792+0000","last_active":"2023-09-30T00:17:51.973474+0000","last_peered":"2023-09-30T00:17:51.973474+0000","last_clean":"2023-09-30T00:17:51.973474+0000","last_became_active":"2023-09-30T00:17:51.965887+0000","last_became_peered":"2023-09-30T00:17:51.965887+0000","last_unstale":"2023-09-30T00:17:51.973474+0000","last_undegraded":"2023-09-30T00:17:51.973474+0000","last_fullsized":"2023-09-30T00:17:51.973474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973636+0000","last_change":"2023-09-30T00:17:51.967175+0000","last_active":"2023-09-30T00:17:51.973636+0000","last_peered":"2023-09-30T00:17:51.973636+0000","last_clean":"2023-09-30T00:17:51.973636+0000","last_became_active":"2023-09-30T00:17:51.966408+0000","last_became_peered":"2023-09-30T00:17:51.966408+0000","last_unstale":"2023-09-30T00:17:51.973636+0000","last_undegraded":"2023-09-30T00:17:51.973636+0000","last_fullsized":"2023-09-30T00:17:51.973636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:47:56.029102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972783+0000","last_change":"2023-09-30T00:17:51.964264+0000","last_active":"2023-09-30T00:17:51.972783+0000","last_peered":"2023-09-30T00:17:51.972783+0000","last_clean":"2023-09-30T00:17:51.972783+0000","last_became_active":"2023-09-30T00:17:51.962740+0000","last_became_peered":"2023-09-30T00:17:51.962740+0000","last_unstale":"2023-09-30T00:17:51.972783+0000","last_undegraded":"2023-09-30T00:17:51.972783+0000","last_fullsized":"2023-09-30T00:17:51.972783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:52:26.654463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975296+0000","last_change":"2023-09-30T00:17:51.967599+0000","last_active":"2023-09-30T00:17:51.975296+0000","last_peered":"2023-09-30T00:17:51.975296+0000","last_clean":"2023-09-30T00:17:51.975296+0000","last_became_active":"2023-09-30T00:17:51.961762+0000","last_became_peered":"2023-09-30T00:17:51.961762+0000","last_unstale":"2023-09-30T00:17:51.975296+0000","last_undegraded":"2023-09-30T00:17:51.975296+0000","last_fullsized":"2023-09-30T00:17:51.975296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:35:31.620523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570714+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:14:31.363561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972400+0000","last_change":"2023-09-30T00:17:51.963624+0000","last_active":"2023-09-30T00:17:51.972400+0000","last_peered":"2023-09-30T00:17:51.972400+0000","last_clean":"2023-09-30T00:17:51.972400+0000","last_became_active":"2023-09-30T00:17:51.963303+0000","last_became_peered":"2023-09-30T00:17:51.963303+0000","last_unstale":"2023-09-30T00:17:51.972400+0000","last_undegraded":"2023-09-30T00:17:51.972400+0000","last_fullsized":"2023-09-30T00:17:51.972400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971954+0000","last_change":"2023-09-30T00:17:51.961620+0000","last_active":"2023-09-30T00:17:51.971954+0000","last_peered":"2023-09-30T00:17:51.971954+0000","last_clean":"2023-09-30T00:17:51.971954+0000","last_became_active":"2023-09-30T00:17:51.958237+0000","last_became_peered":"2023-09-30T00:17:51.958237+0000","last_unstale":"2023-09-30T00:17:51.971954+0000","last_undegraded":"2023-09-30T00:17:51.971954+0000","last_fullsized":"2023-09-30T00:17:51.971954+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974397+0000","last_change":"2023-09-30T00:17:51.974397+0000","last_active":"2023-09-30T00:17:51.974397+0000","last_peered":"2023-09-30T00:17:51.974397+0000","last_clean":"2023-09-30T00:17:51.974397+0000","last_became_active":"2023-09-30T00:17:51.974025+0000","last_became_peered":"2023-09-30T00:17:51.974025+0000","last_unstale":"2023-09-30T00:17:51.974397+0000","last_undegraded":"2023-09-30T00:17:51.974397+0000","last_fullsized":"2023-09-30T00:17:51.974397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:49:07.723751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570140+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:39:04.507564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976147+0000","last_change":"2023-09-30T00:17:51.970985+0000","last_active":"2023-09-30T00:17:51.976147+0000","last_peered":"2023-09-30T00:17:51.976147+0000","last_clean":"2023-09-30T00:17:51.976147+0000","last_became_active":"2023-09-30T00:17:51.966979+0000","last_became_peered":"2023-09-30T00:17:51.966979+0000","last_unstale":"2023-09-30T00:17:51.976147+0000","last_undegraded":"2023-09-30T00:17:51.976147+0000","last_fullsized":"2023-09-30T00:17:51.976147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:09:28.833739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975911+0000","last_change":"2023-09-30T00:17:51.970398+0000","last_active":"2023-09-30T00:17:51.975911+0000","last_peered":"2023-09-30T00:17:51.975911+0000","last_clean":"2023-09-30T00:17:51.975911+0000","last_became_active":"2023-09-30T00:17:51.961675+0000","last_became_peered":"2023-09-30T00:17:51.961675+0000","last_unstale":"2023-09-30T00:17:51.975911+0000","last_undegraded":"2023-09-30T00:17:51.975911+0000","last_fullsized":"2023-09-30T00:17:51.975911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:33:21.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972718+0000","last_change":"2023-09-30T00:17:51.969081+0000","last_active":"2023-09-30T00:17:51.972718+0000","last_peered":"2023-09-30T00:17:51.972718+0000","last_clean":"2023-09-30T00:17:51.972718+0000","last_became_active":"2023-09-30T00:17:51.966447+0000","last_became_peered":"2023-09-30T00:17:51.966447+0000","last_unstale":"2023-09-30T00:17:51.972718+0000","last_undegraded":"2023-09-30T00:17:51.972718+0000","last_fullsized":"2023-09-30T00:17:51.972718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:19:12.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960350+0000","last_change":"2023-09-30T00:17:51.953980+0000","last_active":"2023-09-30T00:17:51.960350+0000","last_peered":"2023-09-30T00:17:51.960350+0000","last_clean":"2023-09-30T00:17:51.960350+0000","last_became_active":"2023-09-30T00:17:51.952191+0000","last_became_peered":"2023-09-30T00:17:51.952191+0000","last_unstale":"2023-09-30T00:17:51.960350+0000","last_undegraded":"2023-09-30T00:17:51.960350+0000","last_fullsized":"2023-09-30T00:17:51.960350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:52:46.774408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973554+0000","last_change":"2023-09-30T00:17:51.969589+0000","last_active":"2023-09-30T00:17:51.973554+0000","last_peered":"2023-09-30T00:17:51.973554+0000","last_clean":"2023-09-30T00:17:51.973554+0000","last_became_active":"2023-09-30T00:17:51.967300+0000","last_became_peered":"2023-09-30T00:17:51.967300+0000","last_unstale":"2023-09-30T00:17:51.973554+0000","last_undegraded":"2023-09-30T00:17:51.973554+0000","last_fullsized":"2023-09-30T00:17:51.973554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:22:57.623135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568987+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:41:47.152317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972047+0000","last_change":"2023-09-30T00:17:51.961502+0000","last_active":"2023-09-30T00:17:51.972047+0000","last_peered":"2023-09-30T00:17:51.972047+0000","last_clean":"2023-09-30T00:17:51.972047+0000","last_became_active":"2023-09-30T00:17:51.961191+0000","last_became_peered":"2023-09-30T00:17:51.961191+0000","last_unstale":"2023-09-30T00:17:51.972047+0000","last_undegraded":"2023-09-30T00:17:51.972047+0000","last_fullsized":"2023-09-30T00:17:51.972047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:05:37.196886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976121+0000","last_change":"2023-09-30T00:17:51.970789+0000","last_active":"2023-09-30T00:17:51.976121+0000","last_peered":"2023-09-30T00:17:51.976121+0000","last_clean":"2023-09-30T00:17:51.976121+0000","last_became_active":"2023-09-30T00:17:51.967616+0000","last_became_peered":"2023-09-30T00:17:51.967616+0000","last_unstale":"2023-09-30T00:17:51.976121+0000","last_undegraded":"2023-09-30T00:17:51.976121+0000","last_fullsized":"2023-09-30T00:17:51.976121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976127+0000","last_change":"2023-09-30T00:17:51.970797+0000","last_active":"2023-09-30T00:17:51.976127+0000","last_peered":"2023-09-30T00:17:51.976127+0000","last_clean":"2023-09-30T00:17:51.976127+0000","last_became_active":"2023-09-30T00:17:51.963382+0000","last_became_peered":"2023-09-30T00:17:51.963382+0000","last_unstale":"2023-09-30T00:17:51.976127+0000","last_undegraded":"2023-09-30T00:17:51.976127+0000","last_fullsized":"2023-09-30T00:17:51.976127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:35:22.077360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972764+0000","last_change":"2023-09-30T00:17:51.963058+0000","last_active":"2023-09-30T00:17:51.972764+0000","last_peered":"2023-09-30T00:17:51.972764+0000","last_clean":"2023-09-30T00:17:51.972764+0000","last_became_active":"2023-09-30T00:17:51.959450+0000","last_became_peered":"2023-09-30T00:17:51.959450+0000","last_unstale":"2023-09-30T00:17:51.972764+0000","last_undegraded":"2023-09-30T00:17:51.972764+0000","last_fullsized":"2023-09-30T00:17:51.972764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:41:36.585190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975442+0000","last_change":"2023-09-30T00:17:51.967581+0000","last_active":"2023-09-30T00:17:51.975442+0000","last_peered":"2023-09-30T00:17:51.975442+0000","last_clean":"2023-09-30T00:17:51.975442+0000","last_became_active":"2023-09-30T00:17:51.961348+0000","last_became_peered":"2023-09-30T00:17:51.961348+0000","last_unstale":"2023-09-30T00:17:51.975442+0000","last_undegraded":"2023-09-30T00:17:51.975442+0000","last_fullsized":"2023-09-30T00:17:51.975442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:11:09.140469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975871+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.975871+0000","last_peered":"2023-09-30T00:17:51.975871+0000","last_clean":"2023-09-30T00:17:51.975871+0000","last_became_active":"2023-09-30T00:17:51.963534+0000","last_became_peered":"2023-09-30T00:17:51.963534+0000","last_unstale":"2023-09-30T00:17:51.975871+0000","last_undegraded":"2023-09-30T00:17:51.975871+0000","last_fullsized":"2023-09-30T00:17:51.975871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:21:49.086269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.569843+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:31.844594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972810+0000","last_change":"2023-09-30T00:17:51.968803+0000","last_active":"2023-09-30T00:17:51.972810+0000","last_peered":"2023-09-30T00:17:51.972810+0000","last_clean":"2023-09-30T00:17:51.972810+0000","last_became_active":"2023-09-30T00:17:51.965833+0000","last_became_peered":"2023-09-30T00:17:51.965833+0000","last_unstale":"2023-09-30T00:17:51.972810+0000","last_undegraded":"2023-09-30T00:17:51.972810+0000","last_fullsized":"2023-09-30T00:17:51.972810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960606+0000","last_change":"2023-09-30T00:17:51.592497+0000","last_active":"2023-09-30T00:17:51.960606+0000","last_peered":"2023-09-30T00:17:51.960606+0000","last_clean":"2023-09-30T00:17:51.960606+0000","last_became_active":"2023-09-30T00:17:51.585128+0000","last_became_peered":"2023-09-30T00:17:51.585128+0000","last_unstale":"2023-09-30T00:17:51.960606+0000","last_undegraded":"2023-09-30T00:17:51.960606+0000","last_fullsized":"2023-09-30T00:17:51.960606+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:46.058014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977744+0000","last_change":"2023-09-30T00:17:51.974520+0000","last_active":"2023-09-30T00:17:51.977744+0000","last_peered":"2023-09-30T00:17:51.977744+0000","last_clean":"2023-09-30T00:17:51.977744+0000","last_became_active":"2023-09-30T00:17:51.974109+0000","last_became_peered":"2023-09-30T00:17:51.974109+0000","last_unstale":"2023-09-30T00:17:51.977744+0000","last_undegraded":"2023-09-30T00:17:51.977744+0000","last_fullsized":"2023-09-30T00:17:51.977744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:25:05.134336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941011+0000","last_change":"2023-09-30T00:17:51.587986+0000","last_active":"2023-09-30T00:17:51.941011+0000","last_peered":"2023-09-30T00:17:51.941011+0000","last_clean":"2023-09-30T00:17:51.941011+0000","last_became_active":"2023-09-30T00:17:51.587724+0000","last_became_peered":"2023-09-30T00:17:51.587724+0000","last_unstale":"2023-09-30T00:17:51.941011+0000","last_undegraded":"2023-09-30T00:17:51.941011+0000","last_fullsized":"2023-09-30T00:17:51.941011+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568369+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:56:13.933771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952700+0000","last_change":"2023-09-30T00:17:51.596511+0000","last_active":"2023-09-30T00:17:51.952700+0000","last_peered":"2023-09-30T00:17:51.952700+0000","last_clean":"2023-09-30T00:17:51.952700+0000","last_became_active":"2023-09-30T00:17:51.596034+0000","last_became_peered":"2023-09-30T00:17:51.596034+0000","last_unstale":"2023-09-30T00:17:51.952700+0000","last_undegraded":"2023-09-30T00:17:51.952700+0000","last_fullsized":"2023-09-30T00:17:51.952700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:44:29.988610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969658+0000","last_change":"2023-09-30T00:17:51.964766+0000","last_active":"2023-09-30T00:17:51.969658+0000","last_peered":"2023-09-30T00:17:51.969658+0000","last_clean":"2023-09-30T00:17:51.969658+0000","last_became_active":"2023-09-30T00:17:51.964496+0000","last_became_peered":"2023-09-30T00:17:51.964496+0000","last_unstale":"2023-09-30T00:17:51.969658+0000","last_undegraded":"2023-09-30T00:17:51.969658+0000","last_fullsized":"2023-09-30T00:17:51.969658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.942081+0000","last_change":"2023-09-30T00:17:51.577022+0000","last_active":"2023-09-30T00:17:51.942081+0000","last_peered":"2023-09-30T00:17:51.942081+0000","last_clean":"2023-09-30T00:17:51.942081+0000","last_became_active":"2023-09-30T00:17:51.576726+0000","last_became_peered":"2023-09-30T00:17:51.576726+0000","last_unstale":"2023-09-30T00:17:51.942081+0000","last_undegraded":"2023-09-30T00:17:51.942081+0000","last_fullsized":"2023-09-30T00:17:51.942081+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972739+0000","last_change":"2023-09-30T00:17:51.969564+0000","last_active":"2023-09-30T00:17:51.972739+0000","last_peered":"2023-09-30T00:17:51.972739+0000","last_clean":"2023-09-30T00:17:51.972739+0000","last_became_active":"2023-09-30T00:17:51.969240+0000","last_became_peered":"2023-09-30T00:17:51.969240+0000","last_unstale":"2023-09-30T00:17:51.972739+0000","last_undegraded":"2023-09-30T00:17:51.972739+0000","last_fullsized":"2023-09-30T00:17:51.972739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:42.701890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.589757+0000","last_change":"2023-09-30T00:17:51.588716+0000","last_active":"2023-09-30T00:17:51.589757+0000","last_peered":"2023-09-30T00:17:51.589757+0000","last_clean":"2023-09-30T00:17:51.589757+0000","last_became_active":"2023-09-30T00:17:51.588454+0000","last_became_peered":"2023-09-30T00:17:51.588454+0000","last_unstale":"2023-09-30T00:17:51.589757+0000","last_undegraded":"2023-09-30T00:17:51.589757+0000","last_fullsized":"2023-09-30T00:17:51.589757+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568152+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:48:09.502104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"18'3","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965653+0000","last_change":"2023-09-30T00:17:51.589988+0000","last_active":"2023-09-30T00:17:51.965653+0000","last_peered":"2023-09-30T00:17:51.965653+0000","last_clean":"2023-09-30T00:17:51.965653+0000","last_became_active":"2023-09-30T00:17:51.589729+0000","last_became_peered":"2023-09-30T00:17:51.589729+0000","last_unstale":"2023-09-30T00:17:51.965653+0000","last_undegraded":"2023-09-30T00:17:51.965653+0000","last_fullsized":"2023-09-30T00:17:51.965653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975239+0000","last_change":"2023-09-30T00:17:51.967245+0000","last_active":"2023-09-30T00:17:51.975239+0000","last_peered":"2023-09-30T00:17:51.975239+0000","last_clean":"2023-09-30T00:17:51.975239+0000","last_became_active":"2023-09-30T00:17:51.963755+0000","last_became_peered":"2023-09-30T00:17:51.963755+0000","last_unstale":"2023-09-30T00:17:51.975239+0000","last_undegraded":"2023-09-30T00:17:51.975239+0000","last_fullsized":"2023-09-30T00:17:51.975239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:57:31.735059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"18'6","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.983073+0000","last_change":"2023-09-30T00:17:51.592307+0000","last_active":"2023-09-30T00:17:51.983073+0000","last_peered":"2023-09-30T00:17:51.983073+0000","last_clean":"2023-09-30T00:17:51.983073+0000","last_became_active":"2023-09-30T00:17:51.591753+0000","last_became_peered":"2023-09-30T00:17:51.591753+0000","last_unstale":"2023-09-30T00:17:51.983073+0000","last_undegraded":"2023-09-30T00:17:51.983073+0000","last_fullsized":"2023-09-30T00:17:51.983073+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:14:46.193926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.967859+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.967859+0000","last_peered":"2023-09-30T00:17:51.967859+0000","last_clean":"2023-09-30T00:17:51.967859+0000","last_became_active":"2023-09-30T00:17:51.967118+0000","last_became_peered":"2023-09-30T00:17:51.967118+0000","last_unstale":"2023-09-30T00:17:51.967859+0000","last_undegraded":"2023-09-30T00:17:51.967859+0000","last_fullsized":"2023-09-30T00:17:51.967859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:50.694814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969722+0000","last_change":"2023-09-30T00:17:51.591516+0000","last_active":"2023-09-30T00:17:51.969722+0000","last_peered":"2023-09-30T00:17:51.969722+0000","last_clean":"2023-09-30T00:17:51.969722+0000","last_became_active":"2023-09-30T00:17:51.591206+0000","last_became_peered":"2023-09-30T00:17:51.591206+0000","last_unstale":"2023-09-30T00:17:51.969722+0000","last_undegraded":"2023-09-30T00:17:51.969722+0000","last_fullsized":"2023-09-30T00:17:51.969722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:22.807978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977754+0000","last_change":"2023-09-30T00:17:51.974628+0000","last_active":"2023-09-30T00:17:51.977754+0000","last_peered":"2023-09-30T00:17:51.977754+0000","last_clean":"2023-09-30T00:17:51.977754+0000","last_became_active":"2023-09-30T00:17:51.974276+0000","last_became_peered":"2023-09-30T00:17:51.974276+0000","last_unstale":"2023-09-30T00:17:51.977754+0000","last_undegraded":"2023-09-30T00:17:51.977754+0000","last_fullsized":"2023-09-30T00:17:51.977754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.590891+0000","last_change":"2023-09-30T00:17:51.589138+0000","last_active":"2023-09-30T00:17:51.590891+0000","last_peered":"2023-09-30T00:17:51.590891+0000","last_clean":"2023-09-30T00:17:51.590891+0000","last_became_active":"2023-09-30T00:17:51.588883+0000","last_became_peered":"2023-09-30T00:17:51.588883+0000","last_unstale":"2023-09-30T00:17:51.590891+0000","last_undegraded":"2023-09-30T00:17:51.590891+0000","last_fullsized":"2023-09-30T00:17:51.590891+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977625+0000","last_change":"2023-09-30T00:17:51.972690+0000","last_active":"2023-09-30T00:17:51.977625+0000","last_peered":"2023-09-30T00:17:51.977625+0000","last_clean":"2023-09-30T00:17:51.977625+0000","last_became_active":"2023-09-30T00:17:51.969872+0000","last_became_peered":"2023-09-30T00:17:51.969872+0000","last_unstale":"2023-09-30T00:17:51.977625+0000","last_undegraded":"2023-09-30T00:17:51.977625+0000","last_fullsized":"2023-09-30T00:17:51.977625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952325+0000","last_change":"2023-09-30T00:17:51.597410+0000","last_active":"2023-09-30T00:17:51.952325+0000","last_peered":"2023-09-30T00:17:51.952325+0000","last_clean":"2023-09-30T00:17:51.952325+0000","last_became_active":"2023-09-30T00:17:51.597168+0000","last_became_peered":"2023-09-30T00:17:51.597168+0000","last_unstale":"2023-09-30T00:17:51.952325+0000","last_undegraded":"2023-09-30T00:17:51.952325+0000","last_fullsized":"2023-09-30T00:17:51.952325+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:44:45.339994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973574+0000","last_change":"2023-09-30T00:17:51.969752+0000","last_active":"2023-09-30T00:17:51.973574+0000","last_peered":"2023-09-30T00:17:51.973574+0000","last_clean":"2023-09-30T00:17:51.973574+0000","last_became_active":"2023-09-30T00:17:51.968993+0000","last_became_peered":"2023-09-30T00:17:51.968993+0000","last_unstale":"2023-09-30T00:17:51.973574+0000","last_undegraded":"2023-09-30T00:17:51.973574+0000","last_fullsized":"2023-09-30T00:17:51.973574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"18'8","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974927+0000","last_change":"2023-09-30T00:17:51.592339+0000","last_active":"2023-09-30T00:17:51.974927+0000","last_peered":"2023-09-30T00:17:51.974927+0000","last_clean":"2023-09-30T00:17:51.974927+0000","last_became_active":"2023-09-30T00:17:51.592134+0000","last_became_peered":"2023-09-30T00:17:51.592134+0000","last_unstale":"2023-09-30T00:17:51.974927+0000","last_undegraded":"2023-09-30T00:17:51.974927+0000","last_fullsized":"2023-09-30T00:17:51.974927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.569711+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:42:10.868448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"18'1","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.944741+0000","last_change":"2023-09-30T00:17:51.577740+0000","last_active":"2023-09-30T00:17:51.944741+0000","last_peered":"2023-09-30T00:17:51.944741+0000","last_clean":"2023-09-30T00:17:51.944741+0000","last_became_active":"2023-09-30T00:17:51.577449+0000","last_became_peered":"2023-09-30T00:17:51.577449+0000","last_unstale":"2023-09-30T00:17:51.944741+0000","last_undegraded":"2023-09-30T00:17:51.944741+0000","last_fullsized":"2023-09-30T00:17:51.944741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977756+0000","last_change":"2023-09-30T00:17:51.974425+0000","last_active":"2023-09-30T00:17:51.977756+0000","last_peered":"2023-09-30T00:17:51.977756+0000","last_clean":"2023-09-30T00:17:51.977756+0000","last_became_active":"2023-09-30T00:17:51.974143+0000","last_became_peered":"2023-09-30T00:17:51.974143+0000","last_unstale":"2023-09-30T00:17:51.977756+0000","last_undegraded":"2023-09-30T00:17:51.977756+0000","last_fullsized":"2023-09-30T00:17:51.977756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:04:54.968664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"18'2","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968091+0000","last_change":"2023-09-30T00:17:51.578710+0000","last_active":"2023-09-30T00:17:51.968091+0000","last_peered":"2023-09-30T00:17:51.968091+0000","last_clean":"2023-09-30T00:17:51.968091+0000","last_became_active":"2023-09-30T00:17:51.578450+0000","last_became_peered":"2023-09-30T00:17:51.578450+0000","last_unstale":"2023-09-30T00:17:51.968091+0000","last_undegraded":"2023-09-30T00:17:51.968091+0000","last_fullsized":"2023-09-30T00:17:51.968091+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.958310+0000","last_change":"2023-09-30T00:17:51.951367+0000","last_active":"2023-09-30T00:17:51.958310+0000","last_peered":"2023-09-30T00:17:51.958310+0000","last_clean":"2023-09-30T00:17:51.958310+0000","last_became_active":"2023-09-30T00:17:51.951031+0000","last_became_peered":"2023-09-30T00:17:51.951031+0000","last_unstale":"2023-09-30T00:17:51.958310+0000","last_undegraded":"2023-09-30T00:17:51.958310+0000","last_fullsized":"2023-09-30T00:17:51.958310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976462+0000","last_change":"2023-09-30T00:17:51.594234+0000","last_active":"2023-09-30T00:17:51.976462+0000","last_peered":"2023-09-30T00:17:51.976462+0000","last_clean":"2023-09-30T00:17:51.976462+0000","last_became_active":"2023-09-30T00:17:51.594049+0000","last_became_peered":"2023-09-30T00:17:51.594049+0000","last_unstale":"2023-09-30T00:17:51.976462+0000","last_undegraded":"2023-09-30T00:17:51.976462+0000","last_fullsized":"2023-09-30T00:17:51.976462+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:53:48.206055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965298+0000","last_change":"2023-09-30T00:17:51.965298+0000","last_active":"2023-09-30T00:17:51.965298+0000","last_peered":"2023-09-30T00:17:51.965298+0000","last_clean":"2023-09-30T00:17:51.965298+0000","last_became_active":"2023-09-30T00:17:51.964965+0000","last_became_peered":"2023-09-30T00:17:51.964965+0000","last_unstale":"2023-09-30T00:17:51.965298+0000","last_undegraded":"2023-09-30T00:17:51.965298+0000","last_fullsized":"2023-09-30T00:17:51.965298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:33:20.583162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.955628+0000","last_change":"2023-09-30T00:17:51.592485+0000","last_active":"2023-09-30T00:17:51.955628+0000","last_peered":"2023-09-30T00:17:51.955628+0000","last_clean":"2023-09-30T00:17:51.955628+0000","last_became_active":"2023-09-30T00:17:51.592054+0000","last_became_peered":"2023-09-30T00:17:51.592054+0000","last_unstale":"2023-09-30T00:17:51.955628+0000","last_undegraded":"2023-09-30T00:17:51.955628+0000","last_fullsized":"2023-09-30T00:17:51.955628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:46:52.872573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952057+0000","last_change":"2023-09-30T00:17:51.952057+0000","last_active":"2023-09-30T00:17:51.952057+0000","last_peered":"2023-09-30T00:17:51.952057+0000","last_clean":"2023-09-30T00:17:51.952057+0000","last_became_active":"2023-09-30T00:17:51.951537+0000","last_became_peered":"2023-09-30T00:17:51.951537+0000","last_unstale":"2023-09-30T00:17:51.952057+0000","last_undegraded":"2023-09-30T00:17:51.952057+0000","last_fullsized":"2023-09-30T00:17:51.952057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:20:25.199612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.578071+0000","last_change":"2023-09-30T00:17:51.578071+0000","last_active":"2023-09-30T00:17:51.578071+0000","last_peered":"2023-09-30T00:17:51.578071+0000","last_clean":"2023-09-30T00:17:51.578071+0000","last_became_active":"2023-09-30T00:17:51.577808+0000","last_became_peered":"2023-09-30T00:17:51.577808+0000","last_unstale":"2023-09-30T00:17:51.578071+0000","last_undegraded":"2023-09-30T00:17:51.578071+0000","last_fullsized":"2023-09-30T00:17:51.578071+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:20.626316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976183+0000","last_change":"2023-09-30T00:17:51.970822+0000","last_active":"2023-09-30T00:17:51.976183+0000","last_peered":"2023-09-30T00:17:51.976183+0000","last_clean":"2023-09-30T00:17:51.976183+0000","last_became_active":"2023-09-30T00:17:51.970520+0000","last_became_peered":"2023-09-30T00:17:51.970520+0000","last_unstale":"2023-09-30T00:17:51.976183+0000","last_undegraded":"2023-09-30T00:17:51.976183+0000","last_fullsized":"2023-09-30T00:17:51.976183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:07.347359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952250+0000","last_change":"2023-09-30T00:17:51.595723+0000","last_active":"2023-09-30T00:17:51.952250+0000","last_peered":"2023-09-30T00:17:51.952250+0000","last_clean":"2023-09-30T00:17:51.952250+0000","last_became_active":"2023-09-30T00:17:51.595473+0000","last_became_peered":"2023-09-30T00:17:51.595473+0000","last_unstale":"2023-09-30T00:17:51.952250+0000","last_undegraded":"2023-09-30T00:17:51.952250+0000","last_fullsized":"2023-09-30T00:17:51.952250+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:06.343890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980343+0000","last_change":"2023-09-30T00:17:51.977010+0000","last_active":"2023-09-30T00:17:51.980343+0000","last_peered":"2023-09-30T00:17:51.980343+0000","last_clean":"2023-09-30T00:17:51.980343+0000","last_became_active":"2023-09-30T00:17:51.976744+0000","last_became_peered":"2023-09-30T00:17:51.976744+0000","last_unstale":"2023-09-30T00:17:51.980343+0000","last_undegraded":"2023-09-30T00:17:51.980343+0000","last_fullsized":"2023-09-30T00:17:51.980343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:02:14.797729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952109+0000","last_change":"2023-09-30T00:17:51.595305+0000","last_active":"2023-09-30T00:17:51.952109+0000","last_peered":"2023-09-30T00:17:51.952109+0000","last_clean":"2023-09-30T00:17:51.952109+0000","last_became_active":"2023-09-30T00:17:51.595042+0000","last_became_peered":"2023-09-30T00:17:51.595042+0000","last_unstale":"2023-09-30T00:17:51.952109+0000","last_undegraded":"2023-09-30T00:17:51.952109+0000","last_fullsized":"2023-09-30T00:17:51.952109+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:54.681232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965816+0000","last_change":"2023-09-30T00:17:51.965816+0000","last_active":"2023-09-30T00:17:51.965816+0000","last_peered":"2023-09-30T00:17:51.965816+0000","last_clean":"2023-09-30T00:17:51.965816+0000","last_became_active":"2023-09-30T00:17:51.965549+0000","last_became_peered":"2023-09-30T00:17:51.965549+0000","last_unstale":"2023-09-30T00:17:51.965816+0000","last_undegraded":"2023-09-30T00:17:51.965816+0000","last_fullsized":"2023-09-30T00:17:51.965816+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:30.526216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.953542+0000","last_change":"2023-09-30T00:17:51.596289+0000","last_active":"2023-09-30T00:17:51.953542+0000","last_peered":"2023-09-30T00:17:51.953542+0000","last_clean":"2023-09-30T00:17:51.953542+0000","last_became_active":"2023-09-30T00:17:51.595689+0000","last_became_peered":"2023-09-30T00:17:51.595689+0000","last_unstale":"2023-09-30T00:17:51.953542+0000","last_undegraded":"2023-09-30T00:17:51.953542+0000","last_fullsized":"2023-09-30T00:17:51.953542+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:46:55.016362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975889+0000","last_change":"2023-09-30T00:17:51.970420+0000","last_active":"2023-09-30T00:17:51.975889+0000","last_peered":"2023-09-30T00:17:51.975889+0000","last_clean":"2023-09-30T00:17:51.975889+0000","last_became_active":"2023-09-30T00:17:51.963999+0000","last_became_peered":"2023-09-30T00:17:51.963999+0000","last_unstale":"2023-09-30T00:17:51.975889+0000","last_undegraded":"2023-09-30T00:17:51.975889+0000","last_fullsized":"2023-09-30T00:17:51.975889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962989+0000","last_change":"2023-09-30T00:17:51.590485+0000","last_active":"2023-09-30T00:17:51.962989+0000","last_peered":"2023-09-30T00:17:51.962989+0000","last_clean":"2023-09-30T00:17:51.962989+0000","last_became_active":"2023-09-30T00:17:51.590210+0000","last_became_peered":"2023-09-30T00:17:51.590210+0000","last_unstale":"2023-09-30T00:17:51.962989+0000","last_undegraded":"2023-09-30T00:17:51.962989+0000","last_fullsized":"2023-09-30T00:17:51.962989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:32.708671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.582012+0000","last_change":"2023-09-30T00:17:45.196688+0000","last_active":"2023-09-30T00:17:51.582012+0000","last_peered":"2023-09-30T00:17:51.582012+0000","last_clean":"2023-09-30T00:17:51.582012+0000","last_became_active":"2023-09-30T00:17:45.196361+0000","last_became_peered":"2023-09-30T00:17:45.196361+0000","last_unstale":"2023-09-30T00:17:51.582012+0000","last_undegraded":"2023-09-30T00:17:51.582012+0000","last_fullsized":"2023-09-30T00:17:51.582012+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977735+0000","last_change":"2023-09-30T00:17:51.974386+0000","last_active":"2023-09-30T00:17:51.977735+0000","last_peered":"2023-09-30T00:17:51.977735+0000","last_clean":"2023-09-30T00:17:51.977735+0000","last_became_active":"2023-09-30T00:17:51.973776+0000","last_became_peered":"2023-09-30T00:17:51.973776+0000","last_unstale":"2023-09-30T00:17:51.977735+0000","last_undegraded":"2023-09-30T00:17:51.977735+0000","last_fullsized":"2023-09-30T00:17:51.977735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:25:21.282436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:17:50.566871+0000","last_change":"2023-09-30T00:17:45.203067+0000","last_active":"2023-09-30T00:17:50.566871+0000","last_peered":"2023-09-30T00:17:50.566871+0000","last_clean":"2023-09-30T00:17:50.566871+0000","last_became_active":"2023-09-30T00:17:45.202823+0000","last_became_peered":"2023-09-30T00:17:45.202823+0000","last_unstale":"2023-09-30T00:17:50.566871+0000","last_undegraded":"2023-09-30T00:17:50.566871+0000","last_fullsized":"2023-09-30T00:17:50.566871+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:26:01.195302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943027+0000","last_change":"2023-09-30T00:17:51.592843+0000","last_active":"2023-09-30T00:17:51.943027+0000","last_peered":"2023-09-30T00:17:51.943027+0000","last_clean":"2023-09-30T00:17:51.943027+0000","last_became_active":"2023-09-30T00:17:51.592601+0000","last_became_peered":"2023-09-30T00:17:51.592601+0000","last_unstale":"2023-09-30T00:17:51.943027+0000","last_undegraded":"2023-09-30T00:17:51.943027+0000","last_fullsized":"2023-09-30T00:17:51.943027+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965286+0000","last_change":"2023-09-30T00:17:51.965286+0000","last_active":"2023-09-30T00:17:51.965286+0000","last_peered":"2023-09-30T00:17:51.965286+0000","last_clean":"2023-09-30T00:17:51.965286+0000","last_became_active":"2023-09-30T00:17:51.964869+0000","last_became_peered":"2023-09-30T00:17:51.964869+0000","last_unstale":"2023-09-30T00:17:51.965286+0000","last_undegraded":"2023-09-30T00:17:51.965286+0000","last_fullsized":"2023-09-30T00:17:51.965286+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:46.192135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:17:50.566873+0000","last_change":"2023-09-30T00:17:45.195737+0000","last_active":"2023-09-30T00:17:50.566873+0000","last_peered":"2023-09-30T00:17:50.566873+0000","last_clean":"2023-09-30T00:17:50.566873+0000","last_became_active":"2023-09-30T00:17:45.195464+0000","last_became_peered":"2023-09-30T00:17:45.195464+0000","last_unstale":"2023-09-30T00:17:50.566873+0000","last_undegraded":"2023-09-30T00:17:50.566873+0000","last_fullsized":"2023-09-30T00:17:50.566873+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:24:48.730978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952759+0000","last_change":"2023-09-30T00:17:47.199132+0000","last_active":"2023-09-30T00:17:51.952759+0000","last_peered":"2023-09-30T00:17:51.952759+0000","last_clean":"2023-09-30T00:17:51.952759+0000","last_became_active":"2023-09-30T00:17:47.198255+0000","last_became_peered":"2023-09-30T00:17:47.198255+0000","last_unstale":"2023-09-30T00:17:51.952759+0000","last_undegraded":"2023-09-30T00:17:51.952759+0000","last_fullsized":"2023-09-30T00:17:51.952759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_clean_scrub_stamp":"2023-09-30T00:17:46.189460+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:09.834201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.575046+0000","last_change":"2023-09-30T00:17:51.575046+0000","last_active":"2023-09-30T00:17:51.575046+0000","last_peered":"2023-09-30T00:17:51.575046+0000","last_clean":"2023-09-30T00:17:51.575046+0000","last_became_active":"2023-09-30T00:17:51.574815+0000","last_became_peered":"2023-09-30T00:17:51.574815+0000","last_unstale":"2023-09-30T00:17:51.575046+0000","last_undegraded":"2023-09-30T00:17:51.575046+0000","last_fullsized":"2023-09-30T00:17:51.575046+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:43:57.892326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973520+0000","last_change":"2023-09-30T00:17:51.969276+0000","last_active":"2023-09-30T00:17:51.973520+0000","last_peered":"2023-09-30T00:17:51.973520+0000","last_clean":"2023-09-30T00:17:51.973520+0000","last_became_active":"2023-09-30T00:17:51.966513+0000","last_became_peered":"2023-09-30T00:17:51.966513+0000","last_unstale":"2023-09-30T00:17:51.973520+0000","last_undegraded":"2023-09-30T00:17:51.973520+0000","last_fullsized":"2023-09-30T00:17:51.973520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.579721+0000","last_change":"2023-09-30T00:17:45.197230+0000","last_active":"2023-09-30T00:17:51.579721+0000","last_peered":"2023-09-30T00:17:51.579721+0000","last_clean":"2023-09-30T00:17:51.579721+0000","last_became_active":"2023-09-30T00:17:45.196528+0000","last_became_peered":"2023-09-30T00:17:45.196528+0000","last_unstale":"2023-09-30T00:17:51.579721+0000","last_undegraded":"2023-09-30T00:17:51.579721+0000","last_fullsized":"2023-09-30T00:17:51.579721+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.575520+0000","last_change":"2023-09-30T00:17:51.575520+0000","last_active":"2023-09-30T00:17:51.575520+0000","last_peered":"2023-09-30T00:17:51.575520+0000","last_clean":"2023-09-30T00:17:51.575520+0000","last_became_active":"2023-09-30T00:17:51.575306+0000","last_became_peered":"2023-09-30T00:17:51.575306+0000","last_unstale":"2023-09-30T00:17:51.575520+0000","last_undegraded":"2023-09-30T00:17:51.575520+0000","last_fullsized":"2023-09-30T00:17:51.575520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:31:32.127076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568586+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:37:36.608579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.940896+0000","last_change":"2023-09-30T00:17:45.196542+0000","last_active":"2023-09-30T00:17:51.940896+0000","last_peered":"2023-09-30T00:17:51.940896+0000","last_clean":"2023-09-30T00:17:51.940896+0000","last_became_active":"2023-09-30T00:17:45.196123+0000","last_became_peered":"2023-09-30T00:17:45.196123+0000","last_unstale":"2023-09-30T00:17:51.940896+0000","last_undegraded":"2023-09-30T00:17:51.940896+0000","last_fullsized":"2023-09-30T00:17:51.940896+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952371+0000","last_change":"2023-09-30T00:17:51.591518+0000","last_active":"2023-09-30T00:17:51.952371+0000","last_peered":"2023-09-30T00:17:51.952371+0000","last_clean":"2023-09-30T00:17:51.952371+0000","last_became_active":"2023-09-30T00:17:51.591253+0000","last_became_peered":"2023-09-30T00:17:51.591253+0000","last_unstale":"2023-09-30T00:17:51.952371+0000","last_undegraded":"2023-09-30T00:17:51.952371+0000","last_fullsized":"2023-09-30T00:17:51.952371+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:52:46.330854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980323+0000","last_change":"2023-09-30T00:17:51.976800+0000","last_active":"2023-09-30T00:17:51.980323+0000","last_peered":"2023-09-30T00:17:51.980323+0000","last_clean":"2023-09-30T00:17:51.980323+0000","last_became_active":"2023-09-30T00:17:51.976602+0000","last_became_peered":"2023-09-30T00:17:51.976602+0000","last_unstale":"2023-09-30T00:17:51.980323+0000","last_undegraded":"2023-09-30T00:17:51.980323+0000","last_fullsized":"2023-09-30T00:17:51.980323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:53:26.550299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952216+0000","last_change":"2023-09-30T00:17:45.207123+0000","last_active":"2023-09-30T00:17:51.952216+0000","last_peered":"2023-09-30T00:17:51.952216+0000","last_clean":"2023-09-30T00:17:51.952216+0000","last_became_active":"2023-09-30T00:17:45.206750+0000","last_became_peered":"2023-09-30T00:17:45.206750+0000","last_unstale":"2023-09-30T00:17:51.952216+0000","last_undegraded":"2023-09-30T00:17:51.952216+0000","last_fullsized":"2023-09-30T00:17:51.952216+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.587209+0000","last_change":"2023-09-30T00:17:51.585515+0000","last_active":"2023-09-30T00:17:51.587209+0000","last_peered":"2023-09-30T00:17:51.587209+0000","last_clean":"2023-09-30T00:17:51.587209+0000","last_became_active":"2023-09-30T00:17:51.585262+0000","last_became_peered":"2023-09-30T00:17:51.585262+0000","last_unstale":"2023-09-30T00:17:51.587209+0000","last_undegraded":"2023-09-30T00:17:51.587209+0000","last_fullsized":"2023-09-30T00:17:51.587209+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:04.914282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568596+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:59.954235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.954829+0000","last_change":"2023-09-30T00:17:45.206971+0000","last_active":"2023-09-30T00:17:51.954829+0000","last_peered":"2023-09-30T00:17:51.954829+0000","last_clean":"2023-09-30T00:17:51.954829+0000","last_became_active":"2023-09-30T00:17:45.206693+0000","last_became_peered":"2023-09-30T00:17:45.206693+0000","last_unstale":"2023-09-30T00:17:51.954829+0000","last_undegraded":"2023-09-30T00:17:51.954829+0000","last_fullsized":"2023-09-30T00:17:51.954829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.575506+0000","last_change":"2023-09-30T00:17:51.575506+0000","last_active":"2023-09-30T00:17:51.575506+0000","last_peered":"2023-09-30T00:17:51.575506+0000","last_clean":"2023-09-30T00:17:51.575506+0000","last_became_active":"2023-09-30T00:17:51.574806+0000","last_became_peered":"2023-09-30T00:17:51.574806+0000","last_unstale":"2023-09-30T00:17:51.575506+0000","last_undegraded":"2023-09-30T00:17:51.575506+0000","last_fullsized":"2023-09-30T00:17:51.575506+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:51.816991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975408+0000","last_change":"2023-09-30T00:17:51.967785+0000","last_active":"2023-09-30T00:17:51.975408+0000","last_peered":"2023-09-30T00:17:51.975408+0000","last_clean":"2023-09-30T00:17:51.975408+0000","last_became_active":"2023-09-30T00:17:51.967419+0000","last_became_peered":"2023-09-30T00:17:51.967419+0000","last_unstale":"2023-09-30T00:17:51.975408+0000","last_undegraded":"2023-09-30T00:17:51.975408+0000","last_fullsized":"2023-09-30T00:17:51.975408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941618+0000","last_change":"2023-09-30T00:17:45.203202+0000","last_active":"2023-09-30T00:17:51.941618+0000","last_peered":"2023-09-30T00:17:51.941618+0000","last_clean":"2023-09-30T00:17:51.941618+0000","last_became_active":"2023-09-30T00:17:45.202879+0000","last_became_peered":"2023-09-30T00:17:45.202879+0000","last_unstale":"2023-09-30T00:17:51.941618+0000","last_undegraded":"2023-09-30T00:17:51.941618+0000","last_fullsized":"2023-09-30T00:17:51.941618+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:50:42.379943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.585090+0000","last_change":"2023-09-30T00:17:51.580771+0000","last_active":"2023-09-30T00:17:51.585090+0000","last_peered":"2023-09-30T00:17:51.585090+0000","last_clean":"2023-09-30T00:17:51.585090+0000","last_became_active":"2023-09-30T00:17:51.580495+0000","last_became_peered":"2023-09-30T00:17:51.580495+0000","last_unstale":"2023-09-30T00:17:51.585090+0000","last_undegraded":"2023-09-30T00:17:51.585090+0000","last_fullsized":"2023-09-30T00:17:51.585090+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971832+0000","last_change":"2023-09-30T00:17:51.957745+0000","last_active":"2023-09-30T00:17:51.971832+0000","last_peered":"2023-09-30T00:17:51.971832+0000","last_clean":"2023-09-30T00:17:51.971832+0000","last_became_active":"2023-09-30T00:17:51.955331+0000","last_became_peered":"2023-09-30T00:17:51.955331+0000","last_unstale":"2023-09-30T00:17:51.971832+0000","last_undegraded":"2023-09-30T00:17:51.971832+0000","last_fullsized":"2023-09-30T00:17:51.971832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:45.393660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973877+0000","last_change":"2023-09-30T00:17:51.596021+0000","last_active":"2023-09-30T00:17:51.973877+0000","last_peered":"2023-09-30T00:17:51.973877+0000","last_clean":"2023-09-30T00:17:51.973877+0000","last_became_active":"2023-09-30T00:17:51.595745+0000","last_became_peered":"2023-09-30T00:17:51.595745+0000","last_unstale":"2023-09-30T00:17:51.973877+0000","last_undegraded":"2023-09-30T00:17:51.973877+0000","last_fullsized":"2023-09-30T00:17:51.973877+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:44.390191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.569794+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:06:14.351937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974567+0000","last_change":"2023-09-30T00:17:51.587276+0000","last_active":"2023-09-30T00:17:51.974567+0000","last_peered":"2023-09-30T00:17:51.974567+0000","last_clean":"2023-09-30T00:17:51.974567+0000","last_became_active":"2023-09-30T00:17:51.586984+0000","last_became_peered":"2023-09-30T00:17:51.586984+0000","last_unstale":"2023-09-30T00:17:51.974567+0000","last_undegraded":"2023-09-30T00:17:51.974567+0000","last_fullsized":"2023-09-30T00:17:51.974567+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.953787+0000","last_change":"2023-09-30T00:17:51.953787+0000","last_active":"2023-09-30T00:17:51.953787+0000","last_peered":"2023-09-30T00:17:51.953787+0000","last_clean":"2023-09-30T00:17:51.953787+0000","last_became_active":"2023-09-30T00:17:51.951805+0000","last_became_peered":"2023-09-30T00:17:51.951805+0000","last_unstale":"2023-09-30T00:17:51.953787+0000","last_undegraded":"2023-09-30T00:17:51.953787+0000","last_fullsized":"2023-09-30T00:17:51.953787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:57:04.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965610+0000","last_change":"2023-09-30T00:17:51.593225+0000","last_active":"2023-09-30T00:17:51.965610+0000","last_peered":"2023-09-30T00:17:51.965610+0000","last_clean":"2023-09-30T00:17:51.965610+0000","last_became_active":"2023-09-30T00:17:51.592913+0000","last_became_peered":"2023-09-30T00:17:51.592913+0000","last_unstale":"2023-09-30T00:17:51.965610+0000","last_undegraded":"2023-09-30T00:17:51.965610+0000","last_fullsized":"2023-09-30T00:17:51.965610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:37:56.776343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.955058+0000","last_change":"2023-09-30T00:17:51.955058+0000","last_active":"2023-09-30T00:17:51.955058+0000","last_peered":"2023-09-30T00:17:51.955058+0000","last_clean":"2023-09-30T00:17:51.955058+0000","last_became_active":"2023-09-30T00:17:51.954787+0000","last_became_peered":"2023-09-30T00:17:51.954787+0000","last_unstale":"2023-09-30T00:17:51.955058+0000","last_undegraded":"2023-09-30T00:17:51.955058+0000","last_fullsized":"2023-09-30T00:17:51.955058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:49:30.148218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.586229+0000","last_change":"2023-09-30T00:17:51.586229+0000","last_active":"2023-09-30T00:17:51.586229+0000","last_peered":"2023-09-30T00:17:51.586229+0000","last_clean":"2023-09-30T00:17:51.586229+0000","last_became_active":"2023-09-30T00:17:51.586027+0000","last_became_peered":"2023-09-30T00:17:51.586027+0000","last_unstale":"2023-09-30T00:17:51.586229+0000","last_undegraded":"2023-09-30T00:17:51.586229+0000","last_fullsized":"2023-09-30T00:17:51.586229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:48:13.817514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962963+0000","last_change":"2023-09-30T00:17:51.959628+0000","last_active":"2023-09-30T00:17:51.962963+0000","last_peered":"2023-09-30T00:17:51.962963+0000","last_clean":"2023-09-30T00:17:51.962963+0000","last_became_active":"2023-09-30T00:17:51.958974+0000","last_became_peered":"2023-09-30T00:17:51.958974+0000","last_unstale":"2023-09-30T00:17:51.962963+0000","last_undegraded":"2023-09-30T00:17:51.962963+0000","last_fullsized":"2023-09-30T00:17:51.962963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:28.604134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"18'4","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960480+0000","last_change":"2023-09-30T00:17:51.590387+0000","last_active":"2023-09-30T00:17:51.960480+0000","last_peered":"2023-09-30T00:17:51.960480+0000","last_clean":"2023-09-30T00:17:51.960480+0000","last_became_active":"2023-09-30T00:17:51.585479+0000","last_became_peered":"2023-09-30T00:17:51.585479+0000","last_unstale":"2023-09-30T00:17:51.960480+0000","last_undegraded":"2023-09-30T00:17:51.960480+0000","last_fullsized":"2023-09-30T00:17:51.960480+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.949418+0000","last_change":"2023-09-30T00:17:51.949418+0000","last_active":"2023-09-30T00:17:51.949418+0000","last_peered":"2023-09-30T00:17:51.949418+0000","last_clean":"2023-09-30T00:17:51.949418+0000","last_became_active":"2023-09-30T00:17:51.949101+0000","last_became_peered":"2023-09-30T00:17:51.949101+0000","last_unstale":"2023-09-30T00:17:51.949418+0000","last_undegraded":"2023-09-30T00:17:51.949418+0000","last_fullsized":"2023-09-30T00:17:51.949418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975274+0000","last_change":"2023-09-30T00:17:51.964155+0000","last_active":"2023-09-30T00:17:51.975274+0000","last_peered":"2023-09-30T00:17:51.975274+0000","last_clean":"2023-09-30T00:17:51.975274+0000","last_became_active":"2023-09-30T00:17:51.958334+0000","last_became_peered":"2023-09-30T00:17:51.958334+0000","last_unstale":"2023-09-30T00:17:51.975274+0000","last_undegraded":"2023-09-30T00:17:51.975274+0000","last_fullsized":"2023-09-30T00:17:51.975274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570057+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:31:59.960264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976104+0000","last_change":"2023-09-30T00:17:51.970791+0000","last_active":"2023-09-30T00:17:51.976104+0000","last_peered":"2023-09-30T00:17:51.976104+0000","last_clean":"2023-09-30T00:17:51.976104+0000","last_became_active":"2023-09-30T00:17:51.964142+0000","last_became_peered":"2023-09-30T00:17:51.964142+0000","last_unstale":"2023-09-30T00:17:51.976104+0000","last_undegraded":"2023-09-30T00:17:51.976104+0000","last_fullsized":"2023-09-30T00:17:51.976104+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568144+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:18:28.873560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570896+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:02:47.475511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672965,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9220,"kb_used_data":764,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362620,"statfs":{"total":96636764160,"available":96627322880,"internally_reserved":0,"allocated":782336,"data_stored":646622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]}]},{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8060,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363780,"statfs":{"total":96636764160,"available":96628510720,"internally_reserved":0,"allocated":184320,"data_stored":55478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8852,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94362988,"statfs":{"total":96636764160,"available":96627699712,"internally_reserved":0,"allocated":733184,"data_stored":634680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8052,"kb_used_data":172,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363788,"statfs":{"total":96636764160,"available":96628518912,"internally_reserved":0,"allocated":176128,"data_stored":55170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.859}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7779999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:17:56.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.37 is not active+clean 2023-09-30T00:17:56.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.570714+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T10:14:31.363561+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.33 is not active+clean 2023-09-30T00:17:56.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.570140+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T09:39:04.507564+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.2d is not active+clean 2023-09-30T00:17:56.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.568987+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T03:41:47.152317+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.26 is not active+clean 2023-09-30T00:17:56.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.569843+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T09:44:31.844594+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1e is not active+clean 2023-09-30T00:17:56.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.568369+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T00:56:13.933771+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.11 is not active+clean 2023-09-30T00:17:56.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.568152+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T03:48:09.502104+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.17 is not active+clean 2023-09-30T00:17:56.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.569711+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T06:42:10.868448+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.7 is not active+clean 2023-09-30T00:17:56.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.568586+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T06:37:36.608579+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.4 is not active+clean 2023-09-30T00:17:56.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.568596+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T00:18:59.954235+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1c is not active+clean 2023-09-30T00:17:56.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.569794+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T08:06:14.351937+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.21 is not active+clean 2023-09-30T00:17:56.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.570057+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T00:31:59.960264+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.23 is not active+clean 2023-09-30T00:17:56.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.568144+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T03:18:28.873560+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.24 is not active+clean 2023-09-30T00:17:56.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'creating+peering', 'last_fresh': '2023-09-30T00:17:51.564162+0000', 'last_change': '2023-09-30T00:17:51.570896+0000', 'last_active': '2023-09-30T00:17:51.564162+0000', 'last_peered': '2023-09-30T00:17:51.564162+0000', 'last_clean': '2023-09-30T00:17:51.564162+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-30T00:17:51.564162+0000', 'last_undegraded': '2023-09-30T00:17:51.564162+0000', 'last_fullsized': '2023-09-30T00:17:51.564162+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'last_clean_scrub_stamp': '2023-09-30T00:17:51.564162+0000', 'objects_scrubbed': 0, 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T01:02:47.475511+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-09-30T00:17:56.717 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-09-30T00:17:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.857+0000 7fd670eb0700 1 Processor -- start 2023-09-30T00:17:56.857 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.857+0000 7fd670eb0700 1 -- start start 2023-09-30T00:17:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd670eb0700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd66c070c50 0x7fd66c0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd670eb0700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 0x7fd66c0a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd670eb0700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd66c13fd70 0x7fd66c1401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.860 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd670eb0700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c0a1310 con 0x7fd66c070c50 2023-09-30T00:17:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd670eb0700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c071020 con 0x7fd66c13fd70 2023-09-30T00:17:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd670eb0700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0b76a0 con 0x7fd66c0b7160 2023-09-30T00:17:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd669d9b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 0x7fd66c0a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd66a59c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd66c070c50 0x7fd66c0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd669d9b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 0x7fd66c0a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39062/0 (socket says 172.21.15.94:39062) 2023-09-30T00:17:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd66a59c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd66c070c50 0x7fd66c0a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50540/0 (socket says 172.21.15.94:50540) 2023-09-30T00:17:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd669d9b700 1 -- 172.21.15.94:0/2398100492 learned_addr learned my addr 172.21.15.94:0/2398100492 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd66ad9d700 1 --2- 172.21.15.94:0/2398100492 >> v2:172.21.15.121:3300/0 conn(0x7fd66c13fd70 0x7fd66c1401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd66a59c700 1 -- 172.21.15.94:0/2398100492 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 msgr2=0x7fd66c0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:56.861 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd66a59c700 1 --2- 172.21.15.94:0/2398100492 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 0x7fd66c0a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd66a59c700 1 -- 172.21.15.94:0/2398100492 >> v2:172.21.15.121:3300/0 conn(0x7fd66c13fd70 msgr2=0x7fd66c1401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd66a59c700 1 --2- 172.21.15.94:0/2398100492 >> v2:172.21.15.121:3300/0 conn(0x7fd66c13fd70 0x7fd66c1401d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd66a59c700 1 -- 172.21.15.94:0/2398100492 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0b7940 con 0x7fd66c070c50 2023-09-30T00:17:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd669d9b700 1 --2- 172.21.15.94:0/2398100492 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 0x7fd66c0a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd66a59c700 1 --2- 172.21.15.94:0/2398100492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd66c070c50 0x7fd66c0a0dd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd66c067c90 tx=0x7fd65c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9661a24d6e6c5d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd66959a700 1 -- 172.21.15.94:0/2398100492 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c011630 con 0x7fd66c070c50 2023-09-30T00:17:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd66959a700 1 -- 172.21.15.94:0/2398100492 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65c0117b0 con 0x7fd66c070c50 2023-09-30T00:17:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.858+0000 7fd66959a700 1 -- 172.21.15.94:0/2398100492 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c011b70 con 0x7fd66c070c50 2023-09-30T00:17:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.859+0000 7fd670eb0700 1 -- 172.21.15.94:0/2398100492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd66c070c50 msgr2=0x7fd66c0a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.859+0000 7fd670eb0700 1 --2- 172.21.15.94:0/2398100492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd66c070c50 0x7fd66c0a0dd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd66c067c90 tx=0x7fd65c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.862 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.859+0000 7fd670eb0700 1 -- 172.21.15.94:0/2398100492 shutdown_connections 2023-09-30T00:17:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.859+0000 7fd670eb0700 1 --2- 172.21.15.94:0/2398100492 >> v2:172.21.15.121:3300/0 conn(0x7fd66c13fd70 0x7fd66c1401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.859+0000 7fd670eb0700 1 --2- 172.21.15.94:0/2398100492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd66c070c50 0x7fd66c0a0dd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.859+0000 7fd670eb0700 1 --2- 172.21.15.94:0/2398100492 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 0x7fd66c0a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.859+0000 7fd670eb0700 1 -- 172.21.15.94:0/2398100492 >> 172.21.15.94:0/2398100492 conn(0x7fd66c06a090 msgr2=0x7fd66c0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.859+0000 7fd670eb0700 1 -- 172.21.15.94:0/2398100492 shutdown_connections 2023-09-30T00:17:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.859+0000 7fd670eb0700 1 -- 172.21.15.94:0/2398100492 wait complete. 2023-09-30T00:17:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.860+0000 7fd670eb0700 1 Processor -- start 2023-09-30T00:17:56.863 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.860+0000 7fd670eb0700 1 -- start start 2023-09-30T00:17:56.866 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.860+0000 7fd670eb0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd66c070c50 0x7fd66c0a5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.866 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.860+0000 7fd670eb0700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 0x7fd66c0ab980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.866 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.860+0000 7fd670eb0700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd66c13fd70 0x7fd66c153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.866 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.860+0000 7fd670eb0700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c141ed0 con 0x7fd66c13fd70 2023-09-30T00:17:56.866 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.860+0000 7fd670eb0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c141b20 con 0x7fd66c070c50 2023-09-30T00:17:56.866 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.860+0000 7fd670eb0700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c141ce0 con 0x7fd66c0b7160 2023-09-30T00:17:56.866 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd669d9b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 0x7fd66c0ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.867 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd66ad9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd66c13fd70 0x7fd66c153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.867 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd669d9b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 0x7fd66c0ab980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39078/0 (socket says 172.21.15.94:39078) 2023-09-30T00:17:56.867 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd66ad9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd66c13fd70 0x7fd66c153f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50546/0 (socket says 172.21.15.94:50546) 2023-09-30T00:17:56.867 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd669d9b700 1 -- 172.21.15.94:0/816249454 learned_addr learned my addr 172.21.15.94:0/816249454 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:56.867 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd66a59c700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd66c070c50 0x7fd66c0a5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.867 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd669d9b700 1 -- 172.21.15.94:0/816249454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd66c070c50 msgr2=0x7fd66c0a5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.867 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd669d9b700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd66c070c50 0x7fd66c0a5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.867 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd669d9b700 1 -- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd66c13fd70 msgr2=0x7fd66c153f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:56.867 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd669d9b700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd66c13fd70 0x7fd66c153f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:56.867 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd669d9b700 1 -- 172.21.15.94:0/816249454 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6600094f0 con 0x7fd66c0b7160 2023-09-30T00:17:56.867 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd66ad9d700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd66c13fd70 0x7fd66c153f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:56.867 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd669d9b700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 0x7fd66c0ab980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd66000ec50 tx=0x7fd660009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd65a7fc700 1 -- 172.21.15.94:0/816249454 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66000c920 con 0x7fd66c0b7160 2023-09-30T00:17:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd65a7fc700 1 -- 172.21.15.94:0/816249454 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66000ca80 con 0x7fd66c0b7160 2023-09-30T00:17:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd670eb0700 1 -- 172.21.15.94:0/816249454 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65c009420 con 0x7fd66c0b7160 2023-09-30T00:17:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd65a7fc700 1 -- 172.21.15.94:0/816249454 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66000ce00 con 0x7fd66c0b7160 2023-09-30T00:17:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.861+0000 7fd670eb0700 1 -- 172.21.15.94:0/816249454 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c0ac080 con 0x7fd66c0b7160 2023-09-30T00:17:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.862+0000 7fd670eb0700 1 -- 172.21.15.94:0/816249454 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66c0ab6c0 con 0x7fd66c0b7160 2023-09-30T00:17:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.863+0000 7fd65a7fc700 1 -- 172.21.15.94:0/816249454 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fd660013680 con 0x7fd66c0b7160 2023-09-30T00:17:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.863+0000 7fd65a7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.864+0000 7fd65a7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.864+0000 7fd65a7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.864+0000 7fd65a7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:56.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.864+0000 7fd65a7fc700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd64c098780 0x7fd64c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:56.869 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.864+0000 7fd66a59c700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd64c098780 0x7fd64c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:56.869 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.864+0000 7fd65a7fc700 1 -- 172.21.15.94:0/816249454 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fd6600c0800 con 0x7fd66c0b7160 2023-09-30T00:17:56.869 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.864+0000 7fd66a59c700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd64c098780 0x7fd64c09ac30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd66c067c90 tx=0x7fd65c017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:56.869 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:56.868+0000 7fd65a7fc700 1 -- 172.21.15.94:0/816249454 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd66008deb0 con 0x7fd66c0b7160 2023-09-30T00:17:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.127+0000 7fd670eb0700 1 -- 172.21.15.94:0/816249454 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fd66c071630 con 0x7fd66c0b7160 2023-09-30T00:17:57.128 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.128+0000 7fd65a7fc700 1 -- 172.21.15.94:0/816249454 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1280 (secure 0 0 0) 0x7fd660021510 con 0x7fd66c0b7160 2023-09-30T00:17:57.128 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:57.129 INFO:teuthology.orchestra.run.smithi094.stdout:{"fsid":"3c891173-587e-4c6f-9c6a-04ec750eedad","health":{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":22,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1696033062,"num_in_osds":4,"osd_in_since":1696033055,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":92},{"state_name":"creating+peering","count":13}],"num_pgs":105,"num_pools":4,"num_objects":22,"data_bytes":592674,"bytes_used":35004416,"bytes_avail":386512052224,"bytes_total":386547056640,"inactive_pgs_ratio":0.12380952388048172,"write_bytes_sec":2518,"read_op_per_sec":0,"write_op_per_sec":8},"fsmap":{"epoch":8,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"a","status":"up:active","gid":4239}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":2,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2023-09-30T00:17:34.348775+0000","services":{}},"progress_events":{}} 2023-09-30T00:17:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.129+0000 7fd670eb0700 10 mgrc shutdown 2023-09-30T00:17:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.130+0000 7fd670eb0700 1 -- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd64c098780 msgr2=0x7fd64c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.130+0000 7fd670eb0700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd64c098780 0x7fd64c09ac30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd66c067c90 tx=0x7fd65c017000 comp rx=0 tx=0).stop 2023-09-30T00:17:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.130+0000 7fd670eb0700 1 -- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 msgr2=0x7fd66c0ab980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.130+0000 7fd670eb0700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 0x7fd66c0ab980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd66000ec50 tx=0x7fd660009250 comp rx=0 tx=0).stop 2023-09-30T00:17:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.130+0000 7fd670eb0700 1 -- 172.21.15.94:0/816249454 shutdown_connections 2023-09-30T00:17:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.130+0000 7fd670eb0700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd64c098780 0x7fd64c09ac30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.130+0000 7fd670eb0700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd66c13fd70 0x7fd66c153f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.130+0000 7fd670eb0700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd66c070c50 0x7fd66c0a5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.130+0000 7fd670eb0700 1 --2- 172.21.15.94:0/816249454 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd66c0b7160 0x7fd66c0ab980 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.130+0000 7fd670eb0700 1 -- 172.21.15.94:0/816249454 >> 172.21.15.94:0/816249454 conn(0x7fd66c06a090 msgr2=0x7fd66c0b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.130+0000 7fd670eb0700 1 -- 172.21.15.94:0/816249454 shutdown_connections 2023-09-30T00:17:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.130+0000 7fd670eb0700 1 -- 172.21.15.94:0/816249454 wait complete. 2023-09-30T00:17:57.142 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-09-30T00:17:57.142 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:17:57.256 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.256+0000 7f32afc16700 1 Processor -- start 2023-09-30T00:17:57.256 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.256+0000 7f32afc16700 1 -- start start 2023-09-30T00:17:57.259 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.256+0000 7f32afc16700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f32a8067890 0x7f32a80b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:57.259 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.256+0000 7f32afc16700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f32a80a5720 0x7f32a80a6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:57.259 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.256+0000 7f32afc16700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80a62b0 con 0x7f32a8067c60 2023-09-30T00:17:57.259 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.256+0000 7f32afc16700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a80a6090 con 0x7f32a8067890 2023-09-30T00:17:57.259 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.256+0000 7f32afc16700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a80a6570 con 0x7f32a80a5720 2023-09-30T00:17:57.259 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ad9b2700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f32a8067890 0x7f32a80b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ad9b2700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f32a8067890 0x7f32a80b4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50560/0 (socket says 172.21.15.94:50560) 2023-09-30T00:17:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ad9b2700 1 -- 172.21.15.94:0/3464999650 learned_addr learned my addr 172.21.15.94:0/3464999650 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ad9b2700 1 -- 172.21.15.94:0/3464999650 >> v2:172.21.15.121:3300/0 conn(0x7f32a80a5720 msgr2=0x7f32a80a6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ae1b3700 1 --2- 172.21.15.94:0/3464999650 >> v2:172.21.15.121:3300/0 conn(0x7f32a80a5720 0x7f32a80a6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ad9b2700 1 --2- 172.21.15.94:0/3464999650 >> v2:172.21.15.121:3300/0 conn(0x7f32a80a5720 0x7f32a80a6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ad9b2700 1 -- 172.21.15.94:0/3464999650 >> v1:172.21.15.121:6789/0 conn(0x7f32a8067c60 legacy=0x7f32a80b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:17:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ad9b2700 1 -- 172.21.15.94:0/3464999650 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a80a6830 con 0x7f32a8067890 2023-09-30T00:17:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ae1b3700 1 --2- 172.21.15.94:0/3464999650 >> v2:172.21.15.121:3300/0 conn(0x7f32a80a5720 0x7f32a80a6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:17:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ad9b2700 1 --2- 172.21.15.94:0/3464999650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a8067890 0x7f32a80b4bd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f32a806dfa0 tx=0x7f32a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6598f0ce2170b601 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ac9b0700 1 -- 172.21.15.94:0/3464999650 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a000b820 con 0x7f32a8067890 2023-09-30T00:17:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ac9b0700 1 -- 172.21.15.94:0/3464999650 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a000b980 con 0x7f32a8067890 2023-09-30T00:17:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.257+0000 7f32ac9b0700 1 -- 172.21.15.94:0/3464999650 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a000bd20 con 0x7f32a8067890 2023-09-30T00:17:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.258+0000 7f32afc16700 1 -- 172.21.15.94:0/3464999650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a8067890 msgr2=0x7f32a80b4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.258+0000 7f32afc16700 1 --2- 172.21.15.94:0/3464999650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a8067890 0x7f32a80b4bd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f32a806dfa0 tx=0x7f32a0009250 comp rx=0 tx=0).stop 2023-09-30T00:17:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.258+0000 7f32afc16700 1 -- 172.21.15.94:0/3464999650 shutdown_connections 2023-09-30T00:17:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.258+0000 7f32afc16700 1 --2- 172.21.15.94:0/3464999650 >> v2:172.21.15.121:3300/0 conn(0x7f32a80a5720 0x7f32a80a6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.258+0000 7f32afc16700 1 --2- 172.21.15.94:0/3464999650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a8067890 0x7f32a80b4bd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.258+0000 7f32afc16700 1 -- 172.21.15.94:0/3464999650 >> 172.21.15.94:0/3464999650 conn(0x7f32a806cb10 msgr2=0x7f32a806cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.258+0000 7f32afc16700 1 -- 172.21.15.94:0/3464999650 shutdown_connections 2023-09-30T00:17:57.262 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.258+0000 7f32afc16700 1 -- 172.21.15.94:0/3464999650 wait complete. 2023-09-30T00:17:57.262 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32afc16700 1 Processor -- start 2023-09-30T00:17:57.262 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32afc16700 1 -- start start 2023-09-30T00:17:57.264 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32afc16700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a8067890 0x7f32a809add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:57.264 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32afc16700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32a8067c60 0x7f32a80a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:57.264 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32afc16700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32a80a5720 0x7f32a80a21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:57.264 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32afc16700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a809ca10 con 0x7f32a8067890 2023-09-30T00:17:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32afc16700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a809b7c0 con 0x7f32a80a5720 2023-09-30T00:17:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32afc16700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a809d800 con 0x7f32a8067c60 2023-09-30T00:17:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32ad9b2700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a8067890 0x7f32a809add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32ad9b2700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a8067890 0x7f32a809add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50570/0 (socket says 172.21.15.94:50570) 2023-09-30T00:17:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32ad9b2700 1 -- 172.21.15.94:0/3505270601 learned_addr learned my addr 172.21.15.94:0/3505270601 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:17:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32ad1b1700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32a8067c60 0x7f32a80a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.259+0000 7f32ae1b3700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32a80a5720 0x7f32a80a21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.260+0000 7f32ad9b2700 1 -- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32a8067c60 msgr2=0x7f32a80a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.260+0000 7f32ad9b2700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32a8067c60 0x7f32a80a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.260+0000 7f32ad9b2700 1 -- 172.21.15.94:0/3505270601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32a80a5720 msgr2=0x7f32a80a21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.260+0000 7f32ad9b2700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32a80a5720 0x7f32a80a21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.260+0000 7f32ad9b2700 1 -- 172.21.15.94:0/3505270601 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a00094f0 con 0x7f32a8067890 2023-09-30T00:17:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.260+0000 7f32ad1b1700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32a8067c60 0x7f32a80a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:17:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.260+0000 7f32ad9b2700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a8067890 0x7f32a809add0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f32a00030c0 tx=0x7f32a0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.260+0000 7f329dffb700 1 -- 172.21.15.94:0/3505270601 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a0011d00 con 0x7f32a8067890 2023-09-30T00:17:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.260+0000 7f329dffb700 1 -- 172.21.15.94:0/3505270601 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a0020070 con 0x7f32a8067890 2023-09-30T00:17:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.260+0000 7f329dffb700 1 -- 172.21.15.94:0/3505270601 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a001b6a0 con 0x7f32a8067890 2023-09-30T00:17:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.260+0000 7f32afc16700 1 -- 172.21.15.94:0/3505270601 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a809dac0 con 0x7f32a8067890 2023-09-30T00:17:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.260+0000 7f32afc16700 1 -- 172.21.15.94:0/3505270601 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a80a37b0 con 0x7f32a8067890 2023-09-30T00:17:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.261+0000 7f32afc16700 1 -- 172.21.15.94:0/3505270601 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a8071630 con 0x7f32a8067890 2023-09-30T00:17:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.262+0000 7f329dffb700 1 -- 172.21.15.94:0/3505270601 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f32a0022030 con 0x7f32a8067890 2023-09-30T00:17:57.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.262+0000 7f329dffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:17:57.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.263+0000 7f329dffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:17:57.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.263+0000 7f329dffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:57.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.263+0000 7f329dffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:17:57.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.263+0000 7f329dffb700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f328c0987b0 0x7f328c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:17:57.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.263+0000 7f32ad1b1700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f328c0987b0 0x7f328c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:17:57.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.263+0000 7f329dffb700 1 -- 172.21.15.94:0/3505270601 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f32a00c2260 con 0x7f32a8067890 2023-09-30T00:17:57.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.263+0000 7f32ad1b1700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f328c0987b0 0x7f328c09ac60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32a80a7510 tx=0x7f3298007ea0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:17:57.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.267+0000 7f329dffb700 1 -- 172.21.15.94:0/3505270601 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f32a008f900 con 0x7f32a8067890 2023-09-30T00:17:57.446 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.445+0000 7f32afc16700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-30T00:17:57.446 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.445+0000 7f32afc16700 1 -- 172.21.15.94:0/3505270601 --> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32a8071630 con 0x7f328c0987b0 2023-09-30T00:17:57.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.452+0000 7f329dffb700 1 -- 172.21.15.94:0/3505270601 <== mgr.4106 v2:172.21.15.94:6808/84242 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+268834 (secure 0 0 0) 0x7f32a8071630 con 0x7f328c0987b0 2023-09-30T00:17:57.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.452+0000 7f329dffb700 20 mgrc handle_command_reply tid 0 r 0 2023-09-30T00:17:57.455 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:17:57.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 10 mgrc shutdown 2023-09-30T00:17:57.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 1 -- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f328c0987b0 msgr2=0x7f328c09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:57.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f328c0987b0 0x7f328c09ac60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32a80a7510 tx=0x7f3298007ea0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 1 -- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a8067890 msgr2=0x7f32a809add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:17:57.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a8067890 0x7f32a809add0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f32a00030c0 tx=0x7f32a0016000 comp rx=0 tx=0).stop 2023-09-30T00:17:57.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 1 -- 172.21.15.94:0/3505270601 shutdown_connections 2023-09-30T00:17:57.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f328c0987b0 0x7f328c09ac60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32a80a5720 0x7f32a80a21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a8067890 0x7f32a809add0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 1 --2- 172.21.15.94:0/3505270601 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32a8067c60 0x7f32a80a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:17:57.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 1 -- 172.21.15.94:0/3505270601 >> 172.21.15.94:0/3505270601 conn(0x7f32a806cb10 msgr2=0x7f32a80630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:17:57.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 1 -- 172.21.15.94:0/3505270601 shutdown_connections 2023-09-30T00:17:57.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:17:57.455+0000 7f32afc16700 1 -- 172.21.15.94:0/3505270601 wait complete. 2023-09-30T00:17:57.459 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2023-09-30T00:17:57.467 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-09-30T00:17:55.851278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":202,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":34184,"kb_used_data":1832,"kb_used_omap":0,"kb_used_meta":32320,"kb_avail":377453176,"statfs":{"total":386547056640,"available":386512052224,"internally_reserved":0,"allocated":1875968,"data_stored":1391950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33095680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.286619"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960193+0000","last_change":"2023-09-30T00:17:51.960193+0000","last_active":"2023-09-30T00:17:51.960193+0000","last_peered":"2023-09-30T00:17:51.960193+0000","last_clean":"2023-09-30T00:17:51.960193+0000","last_became_active":"2023-09-30T00:17:51.959888+0000","last_became_peered":"2023-09-30T00:17:51.959888+0000","last_unstale":"2023-09-30T00:17:51.960193+0000","last_undegraded":"2023-09-30T00:17:51.960193+0000","last_fullsized":"2023-09-30T00:17:51.960193+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:40:42.458928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968577+0000","last_change":"2023-09-30T00:17:51.968577+0000","last_active":"2023-09-30T00:17:51.968577+0000","last_peered":"2023-09-30T00:17:51.968577+0000","last_clean":"2023-09-30T00:17:51.968577+0000","last_became_active":"2023-09-30T00:17:51.968275+0000","last_became_peered":"2023-09-30T00:17:51.968275+0000","last_unstale":"2023-09-30T00:17:51.968577+0000","last_undegraded":"2023-09-30T00:17:51.968577+0000","last_fullsized":"2023-09-30T00:17:51.968577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:19:24.974156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977703+0000","last_change":"2023-09-30T00:17:51.974359+0000","last_active":"2023-09-30T00:17:51.977703+0000","last_peered":"2023-09-30T00:17:51.977703+0000","last_clean":"2023-09-30T00:17:51.977703+0000","last_became_active":"2023-09-30T00:17:51.973789+0000","last_became_peered":"2023-09-30T00:17:51.973789+0000","last_unstale":"2023-09-30T00:17:51.977703+0000","last_undegraded":"2023-09-30T00:17:51.977703+0000","last_fullsized":"2023-09-30T00:17:51.977703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:27.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971859+0000","last_change":"2023-09-30T00:17:51.958127+0000","last_active":"2023-09-30T00:17:51.971859+0000","last_peered":"2023-09-30T00:17:51.971859+0000","last_clean":"2023-09-30T00:17:51.971859+0000","last_became_active":"2023-09-30T00:17:51.957850+0000","last_became_peered":"2023-09-30T00:17:51.957850+0000","last_unstale":"2023-09-30T00:17:51.971859+0000","last_undegraded":"2023-09-30T00:17:51.971859+0000","last_fullsized":"2023-09-30T00:17:51.971859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:48:36.950659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973474+0000","last_change":"2023-09-30T00:17:51.966792+0000","last_active":"2023-09-30T00:17:51.973474+0000","last_peered":"2023-09-30T00:17:51.973474+0000","last_clean":"2023-09-30T00:17:51.973474+0000","last_became_active":"2023-09-30T00:17:51.965887+0000","last_became_peered":"2023-09-30T00:17:51.965887+0000","last_unstale":"2023-09-30T00:17:51.973474+0000","last_undegraded":"2023-09-30T00:17:51.973474+0000","last_fullsized":"2023-09-30T00:17:51.973474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973636+0000","last_change":"2023-09-30T00:17:51.967175+0000","last_active":"2023-09-30T00:17:51.973636+0000","last_peered":"2023-09-30T00:17:51.973636+0000","last_clean":"2023-09-30T00:17:51.973636+0000","last_became_active":"2023-09-30T00:17:51.966408+0000","last_became_peered":"2023-09-30T00:17:51.966408+0000","last_unstale":"2023-09-30T00:17:51.973636+0000","last_undegraded":"2023-09-30T00:17:51.973636+0000","last_fullsized":"2023-09-30T00:17:51.973636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:47:56.029102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972783+0000","last_change":"2023-09-30T00:17:51.964264+0000","last_active":"2023-09-30T00:17:51.972783+0000","last_peered":"2023-09-30T00:17:51.972783+0000","last_clean":"2023-09-30T00:17:51.972783+0000","last_became_active":"2023-09-30T00:17:51.962740+0000","last_became_peered":"2023-09-30T00:17:51.962740+0000","last_unstale":"2023-09-30T00:17:51.972783+0000","last_undegraded":"2023-09-30T00:17:51.972783+0000","last_fullsized":"2023-09-30T00:17:51.972783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:52:26.654463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975296+0000","last_change":"2023-09-30T00:17:51.967599+0000","last_active":"2023-09-30T00:17:51.975296+0000","last_peered":"2023-09-30T00:17:51.975296+0000","last_clean":"2023-09-30T00:17:51.975296+0000","last_became_active":"2023-09-30T00:17:51.961762+0000","last_became_peered":"2023-09-30T00:17:51.961762+0000","last_unstale":"2023-09-30T00:17:51.975296+0000","last_undegraded":"2023-09-30T00:17:51.975296+0000","last_fullsized":"2023-09-30T00:17:51.975296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:35:31.620523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570714+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:14:31.363561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972400+0000","last_change":"2023-09-30T00:17:51.963624+0000","last_active":"2023-09-30T00:17:51.972400+0000","last_peered":"2023-09-30T00:17:51.972400+0000","last_clean":"2023-09-30T00:17:51.972400+0000","last_became_active":"2023-09-30T00:17:51.963303+0000","last_became_peered":"2023-09-30T00:17:51.963303+0000","last_unstale":"2023-09-30T00:17:51.972400+0000","last_undegraded":"2023-09-30T00:17:51.972400+0000","last_fullsized":"2023-09-30T00:17:51.972400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971954+0000","last_change":"2023-09-30T00:17:51.961620+0000","last_active":"2023-09-30T00:17:51.971954+0000","last_peered":"2023-09-30T00:17:51.971954+0000","last_clean":"2023-09-30T00:17:51.971954+0000","last_became_active":"2023-09-30T00:17:51.958237+0000","last_became_peered":"2023-09-30T00:17:51.958237+0000","last_unstale":"2023-09-30T00:17:51.971954+0000","last_undegraded":"2023-09-30T00:17:51.971954+0000","last_fullsized":"2023-09-30T00:17:51.971954+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974397+0000","last_change":"2023-09-30T00:17:51.974397+0000","last_active":"2023-09-30T00:17:51.974397+0000","last_peered":"2023-09-30T00:17:51.974397+0000","last_clean":"2023-09-30T00:17:51.974397+0000","last_became_active":"2023-09-30T00:17:51.974025+0000","last_became_peered":"2023-09-30T00:17:51.974025+0000","last_unstale":"2023-09-30T00:17:51.974397+0000","last_undegraded":"2023-09-30T00:17:51.974397+0000","last_fullsized":"2023-09-30T00:17:51.974397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:49:07.723751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570140+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:39:04.507564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976147+0000","last_change":"2023-09-30T00:17:51.970985+0000","last_active":"2023-09-30T00:17:51.976147+0000","last_peered":"2023-09-30T00:17:51.976147+0000","last_clean":"2023-09-30T00:17:51.976147+0000","last_became_active":"2023-09-30T00:17:51.966979+0000","last_became_peered":"2023-09-30T00:17:51.966979+0000","last_unstale":"2023-09-30T00:17:51.976147+0000","last_undegraded":"2023-09-30T00:17:51.976147+0000","last_fullsized":"2023-09-30T00:17:51.976147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:09:28.833739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975911+0000","last_change":"2023-09-30T00:17:51.970398+0000","last_active":"2023-09-30T00:17:51.975911+0000","last_peered":"2023-09-30T00:17:51.975911+0000","last_clean":"2023-09-30T00:17:51.975911+0000","last_became_active":"2023-09-30T00:17:51.961675+0000","last_became_peered":"2023-09-30T00:17:51.961675+0000","last_unstale":"2023-09-30T00:17:51.975911+0000","last_undegraded":"2023-09-30T00:17:51.975911+0000","last_fullsized":"2023-09-30T00:17:51.975911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:33:21.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972718+0000","last_change":"2023-09-30T00:17:51.969081+0000","last_active":"2023-09-30T00:17:51.972718+0000","last_peered":"2023-09-30T00:17:51.972718+0000","last_clean":"2023-09-30T00:17:51.972718+0000","last_became_active":"2023-09-30T00:17:51.966447+0000","last_became_peered":"2023-09-30T00:17:51.966447+0000","last_unstale":"2023-09-30T00:17:51.972718+0000","last_undegraded":"2023-09-30T00:17:51.972718+0000","last_fullsized":"2023-09-30T00:17:51.972718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:19:12.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960350+0000","last_change":"2023-09-30T00:17:51.953980+0000","last_active":"2023-09-30T00:17:51.960350+0000","last_peered":"2023-09-30T00:17:51.960350+0000","last_clean":"2023-09-30T00:17:51.960350+0000","last_became_active":"2023-09-30T00:17:51.952191+0000","last_became_peered":"2023-09-30T00:17:51.952191+0000","last_unstale":"2023-09-30T00:17:51.960350+0000","last_undegraded":"2023-09-30T00:17:51.960350+0000","last_fullsized":"2023-09-30T00:17:51.960350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:52:46.774408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973554+0000","last_change":"2023-09-30T00:17:51.969589+0000","last_active":"2023-09-30T00:17:51.973554+0000","last_peered":"2023-09-30T00:17:51.973554+0000","last_clean":"2023-09-30T00:17:51.973554+0000","last_became_active":"2023-09-30T00:17:51.967300+0000","last_became_peered":"2023-09-30T00:17:51.967300+0000","last_unstale":"2023-09-30T00:17:51.973554+0000","last_undegraded":"2023-09-30T00:17:51.973554+0000","last_fullsized":"2023-09-30T00:17:51.973554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:22:57.623135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568987+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:41:47.152317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972047+0000","last_change":"2023-09-30T00:17:51.961502+0000","last_active":"2023-09-30T00:17:51.972047+0000","last_peered":"2023-09-30T00:17:51.972047+0000","last_clean":"2023-09-30T00:17:51.972047+0000","last_became_active":"2023-09-30T00:17:51.961191+0000","last_became_peered":"2023-09-30T00:17:51.961191+0000","last_unstale":"2023-09-30T00:17:51.972047+0000","last_undegraded":"2023-09-30T00:17:51.972047+0000","last_fullsized":"2023-09-30T00:17:51.972047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:05:37.196886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976121+0000","last_change":"2023-09-30T00:17:51.970789+0000","last_active":"2023-09-30T00:17:51.976121+0000","last_peered":"2023-09-30T00:17:51.976121+0000","last_clean":"2023-09-30T00:17:51.976121+0000","last_became_active":"2023-09-30T00:17:51.967616+0000","last_became_peered":"2023-09-30T00:17:51.967616+0000","last_unstale":"2023-09-30T00:17:51.976121+0000","last_undegraded":"2023-09-30T00:17:51.976121+0000","last_fullsized":"2023-09-30T00:17:51.976121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976127+0000","last_change":"2023-09-30T00:17:51.970797+0000","last_active":"2023-09-30T00:17:51.976127+0000","last_peered":"2023-09-30T00:17:51.976127+0000","last_clean":"2023-09-30T00:17:51.976127+0000","last_became_active":"2023-09-30T00:17:51.963382+0000","last_became_peered":"2023-09-30T00:17:51.963382+0000","last_unstale":"2023-09-30T00:17:51.976127+0000","last_undegraded":"2023-09-30T00:17:51.976127+0000","last_fullsized":"2023-09-30T00:17:51.976127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:35:22.077360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972764+0000","last_change":"2023-09-30T00:17:51.963058+0000","last_active":"2023-09-30T00:17:51.972764+0000","last_peered":"2023-09-30T00:17:51.972764+0000","last_clean":"2023-09-30T00:17:51.972764+0000","last_became_active":"2023-09-30T00:17:51.959450+0000","last_became_peered":"2023-09-30T00:17:51.959450+0000","last_unstale":"2023-09-30T00:17:51.972764+0000","last_undegraded":"2023-09-30T00:17:51.972764+0000","last_fullsized":"2023-09-30T00:17:51.972764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:41:36.585190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975442+0000","last_change":"2023-09-30T00:17:51.967581+0000","last_active":"2023-09-30T00:17:51.975442+0000","last_peered":"2023-09-30T00:17:51.975442+0000","last_clean":"2023-09-30T00:17:51.975442+0000","last_became_active":"2023-09-30T00:17:51.961348+0000","last_became_peered":"2023-09-30T00:17:51.961348+0000","last_unstale":"2023-09-30T00:17:51.975442+0000","last_undegraded":"2023-09-30T00:17:51.975442+0000","last_fullsized":"2023-09-30T00:17:51.975442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:11:09.140469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975871+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.975871+0000","last_peered":"2023-09-30T00:17:51.975871+0000","last_clean":"2023-09-30T00:17:51.975871+0000","last_became_active":"2023-09-30T00:17:51.963534+0000","last_became_peered":"2023-09-30T00:17:51.963534+0000","last_unstale":"2023-09-30T00:17:51.975871+0000","last_undegraded":"2023-09-30T00:17:51.975871+0000","last_fullsized":"2023-09-30T00:17:51.975871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:21:49.086269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.569843+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:44:31.844594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972810+0000","last_change":"2023-09-30T00:17:51.968803+0000","last_active":"2023-09-30T00:17:51.972810+0000","last_peered":"2023-09-30T00:17:51.972810+0000","last_clean":"2023-09-30T00:17:51.972810+0000","last_became_active":"2023-09-30T00:17:51.965833+0000","last_became_peered":"2023-09-30T00:17:51.965833+0000","last_unstale":"2023-09-30T00:17:51.972810+0000","last_undegraded":"2023-09-30T00:17:51.972810+0000","last_fullsized":"2023-09-30T00:17:51.972810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960606+0000","last_change":"2023-09-30T00:17:51.592497+0000","last_active":"2023-09-30T00:17:51.960606+0000","last_peered":"2023-09-30T00:17:51.960606+0000","last_clean":"2023-09-30T00:17:51.960606+0000","last_became_active":"2023-09-30T00:17:51.585128+0000","last_became_peered":"2023-09-30T00:17:51.585128+0000","last_unstale":"2023-09-30T00:17:51.960606+0000","last_undegraded":"2023-09-30T00:17:51.960606+0000","last_fullsized":"2023-09-30T00:17:51.960606+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:46.058014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977744+0000","last_change":"2023-09-30T00:17:51.974520+0000","last_active":"2023-09-30T00:17:51.977744+0000","last_peered":"2023-09-30T00:17:51.977744+0000","last_clean":"2023-09-30T00:17:51.977744+0000","last_became_active":"2023-09-30T00:17:51.974109+0000","last_became_peered":"2023-09-30T00:17:51.974109+0000","last_unstale":"2023-09-30T00:17:51.977744+0000","last_undegraded":"2023-09-30T00:17:51.977744+0000","last_fullsized":"2023-09-30T00:17:51.977744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:25:05.134336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941011+0000","last_change":"2023-09-30T00:17:51.587986+0000","last_active":"2023-09-30T00:17:51.941011+0000","last_peered":"2023-09-30T00:17:51.941011+0000","last_clean":"2023-09-30T00:17:51.941011+0000","last_became_active":"2023-09-30T00:17:51.587724+0000","last_became_peered":"2023-09-30T00:17:51.587724+0000","last_unstale":"2023-09-30T00:17:51.941011+0000","last_undegraded":"2023-09-30T00:17:51.941011+0000","last_fullsized":"2023-09-30T00:17:51.941011+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568369+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:56:13.933771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952700+0000","last_change":"2023-09-30T00:17:51.596511+0000","last_active":"2023-09-30T00:17:51.952700+0000","last_peered":"2023-09-30T00:17:51.952700+0000","last_clean":"2023-09-30T00:17:51.952700+0000","last_became_active":"2023-09-30T00:17:51.596034+0000","last_became_peered":"2023-09-30T00:17:51.596034+0000","last_unstale":"2023-09-30T00:17:51.952700+0000","last_undegraded":"2023-09-30T00:17:51.952700+0000","last_fullsized":"2023-09-30T00:17:51.952700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:44:29.988610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969658+0000","last_change":"2023-09-30T00:17:51.964766+0000","last_active":"2023-09-30T00:17:51.969658+0000","last_peered":"2023-09-30T00:17:51.969658+0000","last_clean":"2023-09-30T00:17:51.969658+0000","last_became_active":"2023-09-30T00:17:51.964496+0000","last_became_peered":"2023-09-30T00:17:51.964496+0000","last_unstale":"2023-09-30T00:17:51.969658+0000","last_undegraded":"2023-09-30T00:17:51.969658+0000","last_fullsized":"2023-09-30T00:17:51.969658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.942081+0000","last_change":"2023-09-30T00:17:51.577022+0000","last_active":"2023-09-30T00:17:51.942081+0000","last_peered":"2023-09-30T00:17:51.942081+0000","last_clean":"2023-09-30T00:17:51.942081+0000","last_became_active":"2023-09-30T00:17:51.576726+0000","last_became_peered":"2023-09-30T00:17:51.576726+0000","last_unstale":"2023-09-30T00:17:51.942081+0000","last_undegraded":"2023-09-30T00:17:51.942081+0000","last_fullsized":"2023-09-30T00:17:51.942081+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972739+0000","last_change":"2023-09-30T00:17:51.969564+0000","last_active":"2023-09-30T00:17:51.972739+0000","last_peered":"2023-09-30T00:17:51.972739+0000","last_clean":"2023-09-30T00:17:51.972739+0000","last_became_active":"2023-09-30T00:17:51.969240+0000","last_became_peered":"2023-09-30T00:17:51.969240+0000","last_unstale":"2023-09-30T00:17:51.972739+0000","last_undegraded":"2023-09-30T00:17:51.972739+0000","last_fullsized":"2023-09-30T00:17:51.972739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:42.701890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.589757+0000","last_change":"2023-09-30T00:17:51.588716+0000","last_active":"2023-09-30T00:17:51.589757+0000","last_peered":"2023-09-30T00:17:51.589757+0000","last_clean":"2023-09-30T00:17:51.589757+0000","last_became_active":"2023-09-30T00:17:51.588454+0000","last_became_peered":"2023-09-30T00:17:51.588454+0000","last_unstale":"2023-09-30T00:17:51.589757+0000","last_undegraded":"2023-09-30T00:17:51.589757+0000","last_fullsized":"2023-09-30T00:17:51.589757+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568152+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:48:09.502104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"18'3","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965653+0000","last_change":"2023-09-30T00:17:51.589988+0000","last_active":"2023-09-30T00:17:51.965653+0000","last_peered":"2023-09-30T00:17:51.965653+0000","last_clean":"2023-09-30T00:17:51.965653+0000","last_became_active":"2023-09-30T00:17:51.589729+0000","last_became_peered":"2023-09-30T00:17:51.589729+0000","last_unstale":"2023-09-30T00:17:51.965653+0000","last_undegraded":"2023-09-30T00:17:51.965653+0000","last_fullsized":"2023-09-30T00:17:51.965653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975239+0000","last_change":"2023-09-30T00:17:51.967245+0000","last_active":"2023-09-30T00:17:51.975239+0000","last_peered":"2023-09-30T00:17:51.975239+0000","last_clean":"2023-09-30T00:17:51.975239+0000","last_became_active":"2023-09-30T00:17:51.963755+0000","last_became_peered":"2023-09-30T00:17:51.963755+0000","last_unstale":"2023-09-30T00:17:51.975239+0000","last_undegraded":"2023-09-30T00:17:51.975239+0000","last_fullsized":"2023-09-30T00:17:51.975239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:57:31.735059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"18'6","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.983073+0000","last_change":"2023-09-30T00:17:51.592307+0000","last_active":"2023-09-30T00:17:51.983073+0000","last_peered":"2023-09-30T00:17:51.983073+0000","last_clean":"2023-09-30T00:17:51.983073+0000","last_became_active":"2023-09-30T00:17:51.591753+0000","last_became_peered":"2023-09-30T00:17:51.591753+0000","last_unstale":"2023-09-30T00:17:51.983073+0000","last_undegraded":"2023-09-30T00:17:51.983073+0000","last_fullsized":"2023-09-30T00:17:51.983073+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:14:46.193926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.967859+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.967859+0000","last_peered":"2023-09-30T00:17:51.967859+0000","last_clean":"2023-09-30T00:17:51.967859+0000","last_became_active":"2023-09-30T00:17:51.967118+0000","last_became_peered":"2023-09-30T00:17:51.967118+0000","last_unstale":"2023-09-30T00:17:51.967859+0000","last_undegraded":"2023-09-30T00:17:51.967859+0000","last_fullsized":"2023-09-30T00:17:51.967859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:50.694814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969722+0000","last_change":"2023-09-30T00:17:51.591516+0000","last_active":"2023-09-30T00:17:51.969722+0000","last_peered":"2023-09-30T00:17:51.969722+0000","last_clean":"2023-09-30T00:17:51.969722+0000","last_became_active":"2023-09-30T00:17:51.591206+0000","last_became_peered":"2023-09-30T00:17:51.591206+0000","last_unstale":"2023-09-30T00:17:51.969722+0000","last_undegraded":"2023-09-30T00:17:51.969722+0000","last_fullsized":"2023-09-30T00:17:51.969722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:22.807978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977754+0000","last_change":"2023-09-30T00:17:51.974628+0000","last_active":"2023-09-30T00:17:51.977754+0000","last_peered":"2023-09-30T00:17:51.977754+0000","last_clean":"2023-09-30T00:17:51.977754+0000","last_became_active":"2023-09-30T00:17:51.974276+0000","last_became_peered":"2023-09-30T00:17:51.974276+0000","last_unstale":"2023-09-30T00:17:51.977754+0000","last_undegraded":"2023-09-30T00:17:51.977754+0000","last_fullsized":"2023-09-30T00:17:51.977754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.590891+0000","last_change":"2023-09-30T00:17:51.589138+0000","last_active":"2023-09-30T00:17:51.590891+0000","last_peered":"2023-09-30T00:17:51.590891+0000","last_clean":"2023-09-30T00:17:51.590891+0000","last_became_active":"2023-09-30T00:17:51.588883+0000","last_became_peered":"2023-09-30T00:17:51.588883+0000","last_unstale":"2023-09-30T00:17:51.590891+0000","last_undegraded":"2023-09-30T00:17:51.590891+0000","last_fullsized":"2023-09-30T00:17:51.590891+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977625+0000","last_change":"2023-09-30T00:17:51.972690+0000","last_active":"2023-09-30T00:17:51.977625+0000","last_peered":"2023-09-30T00:17:51.977625+0000","last_clean":"2023-09-30T00:17:51.977625+0000","last_became_active":"2023-09-30T00:17:51.969872+0000","last_became_peered":"2023-09-30T00:17:51.969872+0000","last_unstale":"2023-09-30T00:17:51.977625+0000","last_undegraded":"2023-09-30T00:17:51.977625+0000","last_fullsized":"2023-09-30T00:17:51.977625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952325+0000","last_change":"2023-09-30T00:17:51.597410+0000","last_active":"2023-09-30T00:17:51.952325+0000","last_peered":"2023-09-30T00:17:51.952325+0000","last_clean":"2023-09-30T00:17:51.952325+0000","last_became_active":"2023-09-30T00:17:51.597168+0000","last_became_peered":"2023-09-30T00:17:51.597168+0000","last_unstale":"2023-09-30T00:17:51.952325+0000","last_undegraded":"2023-09-30T00:17:51.952325+0000","last_fullsized":"2023-09-30T00:17:51.952325+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:44:45.339994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973574+0000","last_change":"2023-09-30T00:17:51.969752+0000","last_active":"2023-09-30T00:17:51.973574+0000","last_peered":"2023-09-30T00:17:51.973574+0000","last_clean":"2023-09-30T00:17:51.973574+0000","last_became_active":"2023-09-30T00:17:51.968993+0000","last_became_peered":"2023-09-30T00:17:51.968993+0000","last_unstale":"2023-09-30T00:17:51.973574+0000","last_undegraded":"2023-09-30T00:17:51.973574+0000","last_fullsized":"2023-09-30T00:17:51.973574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"18'8","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974927+0000","last_change":"2023-09-30T00:17:51.592339+0000","last_active":"2023-09-30T00:17:51.974927+0000","last_peered":"2023-09-30T00:17:51.974927+0000","last_clean":"2023-09-30T00:17:51.974927+0000","last_became_active":"2023-09-30T00:17:51.592134+0000","last_became_peered":"2023-09-30T00:17:51.592134+0000","last_unstale":"2023-09-30T00:17:51.974927+0000","last_undegraded":"2023-09-30T00:17:51.974927+0000","last_fullsized":"2023-09-30T00:17:51.974927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.569711+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:42:10.868448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"18'1","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.944741+0000","last_change":"2023-09-30T00:17:51.577740+0000","last_active":"2023-09-30T00:17:51.944741+0000","last_peered":"2023-09-30T00:17:51.944741+0000","last_clean":"2023-09-30T00:17:51.944741+0000","last_became_active":"2023-09-30T00:17:51.577449+0000","last_became_peered":"2023-09-30T00:17:51.577449+0000","last_unstale":"2023-09-30T00:17:51.944741+0000","last_undegraded":"2023-09-30T00:17:51.944741+0000","last_fullsized":"2023-09-30T00:17:51.944741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977756+0000","last_change":"2023-09-30T00:17:51.974425+0000","last_active":"2023-09-30T00:17:51.977756+0000","last_peered":"2023-09-30T00:17:51.977756+0000","last_clean":"2023-09-30T00:17:51.977756+0000","last_became_active":"2023-09-30T00:17:51.974143+0000","last_became_peered":"2023-09-30T00:17:51.974143+0000","last_unstale":"2023-09-30T00:17:51.977756+0000","last_undegraded":"2023-09-30T00:17:51.977756+0000","last_fullsized":"2023-09-30T00:17:51.977756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:04:54.968664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"18'2","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968091+0000","last_change":"2023-09-30T00:17:51.578710+0000","last_active":"2023-09-30T00:17:51.968091+0000","last_peered":"2023-09-30T00:17:51.968091+0000","last_clean":"2023-09-30T00:17:51.968091+0000","last_became_active":"2023-09-30T00:17:51.578450+0000","last_became_peered":"2023-09-30T00:17:51.578450+0000","last_unstale":"2023-09-30T00:17:51.968091+0000","last_undegraded":"2023-09-30T00:17:51.968091+0000","last_fullsized":"2023-09-30T00:17:51.968091+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.958310+0000","last_change":"2023-09-30T00:17:51.951367+0000","last_active":"2023-09-30T00:17:51.958310+0000","last_peered":"2023-09-30T00:17:51.958310+0000","last_clean":"2023-09-30T00:17:51.958310+0000","last_became_active":"2023-09-30T00:17:51.951031+0000","last_became_peered":"2023-09-30T00:17:51.951031+0000","last_unstale":"2023-09-30T00:17:51.958310+0000","last_undegraded":"2023-09-30T00:17:51.958310+0000","last_fullsized":"2023-09-30T00:17:51.958310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976462+0000","last_change":"2023-09-30T00:17:51.594234+0000","last_active":"2023-09-30T00:17:51.976462+0000","last_peered":"2023-09-30T00:17:51.976462+0000","last_clean":"2023-09-30T00:17:51.976462+0000","last_became_active":"2023-09-30T00:17:51.594049+0000","last_became_peered":"2023-09-30T00:17:51.594049+0000","last_unstale":"2023-09-30T00:17:51.976462+0000","last_undegraded":"2023-09-30T00:17:51.976462+0000","last_fullsized":"2023-09-30T00:17:51.976462+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:53:48.206055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965298+0000","last_change":"2023-09-30T00:17:51.965298+0000","last_active":"2023-09-30T00:17:51.965298+0000","last_peered":"2023-09-30T00:17:51.965298+0000","last_clean":"2023-09-30T00:17:51.965298+0000","last_became_active":"2023-09-30T00:17:51.964965+0000","last_became_peered":"2023-09-30T00:17:51.964965+0000","last_unstale":"2023-09-30T00:17:51.965298+0000","last_undegraded":"2023-09-30T00:17:51.965298+0000","last_fullsized":"2023-09-30T00:17:51.965298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:33:20.583162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.955628+0000","last_change":"2023-09-30T00:17:51.592485+0000","last_active":"2023-09-30T00:17:51.955628+0000","last_peered":"2023-09-30T00:17:51.955628+0000","last_clean":"2023-09-30T00:17:51.955628+0000","last_became_active":"2023-09-30T00:17:51.592054+0000","last_became_peered":"2023-09-30T00:17:51.592054+0000","last_unstale":"2023-09-30T00:17:51.955628+0000","last_undegraded":"2023-09-30T00:17:51.955628+0000","last_fullsized":"2023-09-30T00:17:51.955628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:46:52.872573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952057+0000","last_change":"2023-09-30T00:17:51.952057+0000","last_active":"2023-09-30T00:17:51.952057+0000","last_peered":"2023-09-30T00:17:51.952057+0000","last_clean":"2023-09-30T00:17:51.952057+0000","last_became_active":"2023-09-30T00:17:51.951537+0000","last_became_peered":"2023-09-30T00:17:51.951537+0000","last_unstale":"2023-09-30T00:17:51.952057+0000","last_undegraded":"2023-09-30T00:17:51.952057+0000","last_fullsized":"2023-09-30T00:17:51.952057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:20:25.199612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.578071+0000","last_change":"2023-09-30T00:17:51.578071+0000","last_active":"2023-09-30T00:17:51.578071+0000","last_peered":"2023-09-30T00:17:51.578071+0000","last_clean":"2023-09-30T00:17:51.578071+0000","last_became_active":"2023-09-30T00:17:51.577808+0000","last_became_peered":"2023-09-30T00:17:51.577808+0000","last_unstale":"2023-09-30T00:17:51.578071+0000","last_undegraded":"2023-09-30T00:17:51.578071+0000","last_fullsized":"2023-09-30T00:17:51.578071+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:20.626316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976183+0000","last_change":"2023-09-30T00:17:51.970822+0000","last_active":"2023-09-30T00:17:51.976183+0000","last_peered":"2023-09-30T00:17:51.976183+0000","last_clean":"2023-09-30T00:17:51.976183+0000","last_became_active":"2023-09-30T00:17:51.970520+0000","last_became_peered":"2023-09-30T00:17:51.970520+0000","last_unstale":"2023-09-30T00:17:51.976183+0000","last_undegraded":"2023-09-30T00:17:51.976183+0000","last_fullsized":"2023-09-30T00:17:51.976183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:07.347359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952250+0000","last_change":"2023-09-30T00:17:51.595723+0000","last_active":"2023-09-30T00:17:51.952250+0000","last_peered":"2023-09-30T00:17:51.952250+0000","last_clean":"2023-09-30T00:17:51.952250+0000","last_became_active":"2023-09-30T00:17:51.595473+0000","last_became_peered":"2023-09-30T00:17:51.595473+0000","last_unstale":"2023-09-30T00:17:51.952250+0000","last_undegraded":"2023-09-30T00:17:51.952250+0000","last_fullsized":"2023-09-30T00:17:51.952250+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:06.343890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980343+0000","last_change":"2023-09-30T00:17:51.977010+0000","last_active":"2023-09-30T00:17:51.980343+0000","last_peered":"2023-09-30T00:17:51.980343+0000","last_clean":"2023-09-30T00:17:51.980343+0000","last_became_active":"2023-09-30T00:17:51.976744+0000","last_became_peered":"2023-09-30T00:17:51.976744+0000","last_unstale":"2023-09-30T00:17:51.980343+0000","last_undegraded":"2023-09-30T00:17:51.980343+0000","last_fullsized":"2023-09-30T00:17:51.980343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:02:14.797729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952109+0000","last_change":"2023-09-30T00:17:51.595305+0000","last_active":"2023-09-30T00:17:51.952109+0000","last_peered":"2023-09-30T00:17:51.952109+0000","last_clean":"2023-09-30T00:17:51.952109+0000","last_became_active":"2023-09-30T00:17:51.595042+0000","last_became_peered":"2023-09-30T00:17:51.595042+0000","last_unstale":"2023-09-30T00:17:51.952109+0000","last_undegraded":"2023-09-30T00:17:51.952109+0000","last_fullsized":"2023-09-30T00:17:51.952109+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:54.681232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965816+0000","last_change":"2023-09-30T00:17:51.965816+0000","last_active":"2023-09-30T00:17:51.965816+0000","last_peered":"2023-09-30T00:17:51.965816+0000","last_clean":"2023-09-30T00:17:51.965816+0000","last_became_active":"2023-09-30T00:17:51.965549+0000","last_became_peered":"2023-09-30T00:17:51.965549+0000","last_unstale":"2023-09-30T00:17:51.965816+0000","last_undegraded":"2023-09-30T00:17:51.965816+0000","last_fullsized":"2023-09-30T00:17:51.965816+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:30.526216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.953542+0000","last_change":"2023-09-30T00:17:51.596289+0000","last_active":"2023-09-30T00:17:51.953542+0000","last_peered":"2023-09-30T00:17:51.953542+0000","last_clean":"2023-09-30T00:17:51.953542+0000","last_became_active":"2023-09-30T00:17:51.595689+0000","last_became_peered":"2023-09-30T00:17:51.595689+0000","last_unstale":"2023-09-30T00:17:51.953542+0000","last_undegraded":"2023-09-30T00:17:51.953542+0000","last_fullsized":"2023-09-30T00:17:51.953542+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:46:55.016362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975889+0000","last_change":"2023-09-30T00:17:51.970420+0000","last_active":"2023-09-30T00:17:51.975889+0000","last_peered":"2023-09-30T00:17:51.975889+0000","last_clean":"2023-09-30T00:17:51.975889+0000","last_became_active":"2023-09-30T00:17:51.963999+0000","last_became_peered":"2023-09-30T00:17:51.963999+0000","last_unstale":"2023-09-30T00:17:51.975889+0000","last_undegraded":"2023-09-30T00:17:51.975889+0000","last_fullsized":"2023-09-30T00:17:51.975889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962989+0000","last_change":"2023-09-30T00:17:51.590485+0000","last_active":"2023-09-30T00:17:51.962989+0000","last_peered":"2023-09-30T00:17:51.962989+0000","last_clean":"2023-09-30T00:17:51.962989+0000","last_became_active":"2023-09-30T00:17:51.590210+0000","last_became_peered":"2023-09-30T00:17:51.590210+0000","last_unstale":"2023-09-30T00:17:51.962989+0000","last_undegraded":"2023-09-30T00:17:51.962989+0000","last_fullsized":"2023-09-30T00:17:51.962989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:32.708671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.582012+0000","last_change":"2023-09-30T00:17:45.196688+0000","last_active":"2023-09-30T00:17:51.582012+0000","last_peered":"2023-09-30T00:17:51.582012+0000","last_clean":"2023-09-30T00:17:51.582012+0000","last_became_active":"2023-09-30T00:17:45.196361+0000","last_became_peered":"2023-09-30T00:17:45.196361+0000","last_unstale":"2023-09-30T00:17:51.582012+0000","last_undegraded":"2023-09-30T00:17:51.582012+0000","last_fullsized":"2023-09-30T00:17:51.582012+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977735+0000","last_change":"2023-09-30T00:17:51.974386+0000","last_active":"2023-09-30T00:17:51.977735+0000","last_peered":"2023-09-30T00:17:51.977735+0000","last_clean":"2023-09-30T00:17:51.977735+0000","last_became_active":"2023-09-30T00:17:51.973776+0000","last_became_peered":"2023-09-30T00:17:51.973776+0000","last_unstale":"2023-09-30T00:17:51.977735+0000","last_undegraded":"2023-09-30T00:17:51.977735+0000","last_fullsized":"2023-09-30T00:17:51.977735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:25:21.282436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:17:50.566871+0000","last_change":"2023-09-30T00:17:45.203067+0000","last_active":"2023-09-30T00:17:50.566871+0000","last_peered":"2023-09-30T00:17:50.566871+0000","last_clean":"2023-09-30T00:17:50.566871+0000","last_became_active":"2023-09-30T00:17:45.202823+0000","last_became_peered":"2023-09-30T00:17:45.202823+0000","last_unstale":"2023-09-30T00:17:50.566871+0000","last_undegraded":"2023-09-30T00:17:50.566871+0000","last_fullsized":"2023-09-30T00:17:50.566871+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:26:01.195302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943027+0000","last_change":"2023-09-30T00:17:51.592843+0000","last_active":"2023-09-30T00:17:51.943027+0000","last_peered":"2023-09-30T00:17:51.943027+0000","last_clean":"2023-09-30T00:17:51.943027+0000","last_became_active":"2023-09-30T00:17:51.592601+0000","last_became_peered":"2023-09-30T00:17:51.592601+0000","last_unstale":"2023-09-30T00:17:51.943027+0000","last_undegraded":"2023-09-30T00:17:51.943027+0000","last_fullsized":"2023-09-30T00:17:51.943027+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965286+0000","last_change":"2023-09-30T00:17:51.965286+0000","last_active":"2023-09-30T00:17:51.965286+0000","last_peered":"2023-09-30T00:17:51.965286+0000","last_clean":"2023-09-30T00:17:51.965286+0000","last_became_active":"2023-09-30T00:17:51.964869+0000","last_became_peered":"2023-09-30T00:17:51.964869+0000","last_unstale":"2023-09-30T00:17:51.965286+0000","last_undegraded":"2023-09-30T00:17:51.965286+0000","last_fullsized":"2023-09-30T00:17:51.965286+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:46.192135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:17:50.566873+0000","last_change":"2023-09-30T00:17:45.195737+0000","last_active":"2023-09-30T00:17:50.566873+0000","last_peered":"2023-09-30T00:17:50.566873+0000","last_clean":"2023-09-30T00:17:50.566873+0000","last_became_active":"2023-09-30T00:17:45.195464+0000","last_became_peered":"2023-09-30T00:17:45.195464+0000","last_unstale":"2023-09-30T00:17:50.566873+0000","last_undegraded":"2023-09-30T00:17:50.566873+0000","last_fullsized":"2023-09-30T00:17:50.566873+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:24:48.730978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952759+0000","last_change":"2023-09-30T00:17:47.199132+0000","last_active":"2023-09-30T00:17:51.952759+0000","last_peered":"2023-09-30T00:17:51.952759+0000","last_clean":"2023-09-30T00:17:51.952759+0000","last_became_active":"2023-09-30T00:17:47.198255+0000","last_became_peered":"2023-09-30T00:17:47.198255+0000","last_unstale":"2023-09-30T00:17:51.952759+0000","last_undegraded":"2023-09-30T00:17:51.952759+0000","last_fullsized":"2023-09-30T00:17:51.952759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_clean_scrub_stamp":"2023-09-30T00:17:46.189460+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:09.834201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.575046+0000","last_change":"2023-09-30T00:17:51.575046+0000","last_active":"2023-09-30T00:17:51.575046+0000","last_peered":"2023-09-30T00:17:51.575046+0000","last_clean":"2023-09-30T00:17:51.575046+0000","last_became_active":"2023-09-30T00:17:51.574815+0000","last_became_peered":"2023-09-30T00:17:51.574815+0000","last_unstale":"2023-09-30T00:17:51.575046+0000","last_undegraded":"2023-09-30T00:17:51.575046+0000","last_fullsized":"2023-09-30T00:17:51.575046+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:43:57.892326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973520+0000","last_change":"2023-09-30T00:17:51.969276+0000","last_active":"2023-09-30T00:17:51.973520+0000","last_peered":"2023-09-30T00:17:51.973520+0000","last_clean":"2023-09-30T00:17:51.973520+0000","last_became_active":"2023-09-30T00:17:51.966513+0000","last_became_peered":"2023-09-30T00:17:51.966513+0000","last_unstale":"2023-09-30T00:17:51.973520+0000","last_undegraded":"2023-09-30T00:17:51.973520+0000","last_fullsized":"2023-09-30T00:17:51.973520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.579721+0000","last_change":"2023-09-30T00:17:45.197230+0000","last_active":"2023-09-30T00:17:51.579721+0000","last_peered":"2023-09-30T00:17:51.579721+0000","last_clean":"2023-09-30T00:17:51.579721+0000","last_became_active":"2023-09-30T00:17:45.196528+0000","last_became_peered":"2023-09-30T00:17:45.196528+0000","last_unstale":"2023-09-30T00:17:51.579721+0000","last_undegraded":"2023-09-30T00:17:51.579721+0000","last_fullsized":"2023-09-30T00:17:51.579721+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.575520+0000","last_change":"2023-09-30T00:17:51.575520+0000","last_active":"2023-09-30T00:17:51.575520+0000","last_peered":"2023-09-30T00:17:51.575520+0000","last_clean":"2023-09-30T00:17:51.575520+0000","last_became_active":"2023-09-30T00:17:51.575306+0000","last_became_peered":"2023-09-30T00:17:51.575306+0000","last_unstale":"2023-09-30T00:17:51.575520+0000","last_undegraded":"2023-09-30T00:17:51.575520+0000","last_fullsized":"2023-09-30T00:17:51.575520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:31:32.127076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568586+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:37:36.608579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.940896+0000","last_change":"2023-09-30T00:17:45.196542+0000","last_active":"2023-09-30T00:17:51.940896+0000","last_peered":"2023-09-30T00:17:51.940896+0000","last_clean":"2023-09-30T00:17:51.940896+0000","last_became_active":"2023-09-30T00:17:45.196123+0000","last_became_peered":"2023-09-30T00:17:45.196123+0000","last_unstale":"2023-09-30T00:17:51.940896+0000","last_undegraded":"2023-09-30T00:17:51.940896+0000","last_fullsized":"2023-09-30T00:17:51.940896+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952371+0000","last_change":"2023-09-30T00:17:51.591518+0000","last_active":"2023-09-30T00:17:51.952371+0000","last_peered":"2023-09-30T00:17:51.952371+0000","last_clean":"2023-09-30T00:17:51.952371+0000","last_became_active":"2023-09-30T00:17:51.591253+0000","last_became_peered":"2023-09-30T00:17:51.591253+0000","last_unstale":"2023-09-30T00:17:51.952371+0000","last_undegraded":"2023-09-30T00:17:51.952371+0000","last_fullsized":"2023-09-30T00:17:51.952371+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:52:46.330854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980323+0000","last_change":"2023-09-30T00:17:51.976800+0000","last_active":"2023-09-30T00:17:51.980323+0000","last_peered":"2023-09-30T00:17:51.980323+0000","last_clean":"2023-09-30T00:17:51.980323+0000","last_became_active":"2023-09-30T00:17:51.976602+0000","last_became_peered":"2023-09-30T00:17:51.976602+0000","last_unstale":"2023-09-30T00:17:51.980323+0000","last_undegraded":"2023-09-30T00:17:51.980323+0000","last_fullsized":"2023-09-30T00:17:51.980323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:53:26.550299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952216+0000","last_change":"2023-09-30T00:17:45.207123+0000","last_active":"2023-09-30T00:17:51.952216+0000","last_peered":"2023-09-30T00:17:51.952216+0000","last_clean":"2023-09-30T00:17:51.952216+0000","last_became_active":"2023-09-30T00:17:45.206750+0000","last_became_peered":"2023-09-30T00:17:45.206750+0000","last_unstale":"2023-09-30T00:17:51.952216+0000","last_undegraded":"2023-09-30T00:17:51.952216+0000","last_fullsized":"2023-09-30T00:17:51.952216+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.587209+0000","last_change":"2023-09-30T00:17:51.585515+0000","last_active":"2023-09-30T00:17:51.587209+0000","last_peered":"2023-09-30T00:17:51.587209+0000","last_clean":"2023-09-30T00:17:51.587209+0000","last_became_active":"2023-09-30T00:17:51.585262+0000","last_became_peered":"2023-09-30T00:17:51.585262+0000","last_unstale":"2023-09-30T00:17:51.587209+0000","last_undegraded":"2023-09-30T00:17:51.587209+0000","last_fullsized":"2023-09-30T00:17:51.587209+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:04.914282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568596+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:59.954235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.954829+0000","last_change":"2023-09-30T00:17:45.206971+0000","last_active":"2023-09-30T00:17:51.954829+0000","last_peered":"2023-09-30T00:17:51.954829+0000","last_clean":"2023-09-30T00:17:51.954829+0000","last_became_active":"2023-09-30T00:17:45.206693+0000","last_became_peered":"2023-09-30T00:17:45.206693+0000","last_unstale":"2023-09-30T00:17:51.954829+0000","last_undegraded":"2023-09-30T00:17:51.954829+0000","last_fullsized":"2023-09-30T00:17:51.954829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.575506+0000","last_change":"2023-09-30T00:17:51.575506+0000","last_active":"2023-09-30T00:17:51.575506+0000","last_peered":"2023-09-30T00:17:51.575506+0000","last_clean":"2023-09-30T00:17:51.575506+0000","last_became_active":"2023-09-30T00:17:51.574806+0000","last_became_peered":"2023-09-30T00:17:51.574806+0000","last_unstale":"2023-09-30T00:17:51.575506+0000","last_undegraded":"2023-09-30T00:17:51.575506+0000","last_fullsized":"2023-09-30T00:17:51.575506+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:51.816991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975408+0000","last_change":"2023-09-30T00:17:51.967785+0000","last_active":"2023-09-30T00:17:51.975408+0000","last_peered":"2023-09-30T00:17:51.975408+0000","last_clean":"2023-09-30T00:17:51.975408+0000","last_became_active":"2023-09-30T00:17:51.967419+0000","last_became_peered":"2023-09-30T00:17:51.967419+0000","last_unstale":"2023-09-30T00:17:51.975408+0000","last_undegraded":"2023-09-30T00:17:51.975408+0000","last_fullsized":"2023-09-30T00:17:51.975408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941618+0000","last_change":"2023-09-30T00:17:45.203202+0000","last_active":"2023-09-30T00:17:51.941618+0000","last_peered":"2023-09-30T00:17:51.941618+0000","last_clean":"2023-09-30T00:17:51.941618+0000","last_became_active":"2023-09-30T00:17:45.202879+0000","last_became_peered":"2023-09-30T00:17:45.202879+0000","last_unstale":"2023-09-30T00:17:51.941618+0000","last_undegraded":"2023-09-30T00:17:51.941618+0000","last_fullsized":"2023-09-30T00:17:51.941618+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:50:42.379943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.585090+0000","last_change":"2023-09-30T00:17:51.580771+0000","last_active":"2023-09-30T00:17:51.585090+0000","last_peered":"2023-09-30T00:17:51.585090+0000","last_clean":"2023-09-30T00:17:51.585090+0000","last_became_active":"2023-09-30T00:17:51.580495+0000","last_became_peered":"2023-09-30T00:17:51.580495+0000","last_unstale":"2023-09-30T00:17:51.585090+0000","last_undegraded":"2023-09-30T00:17:51.585090+0000","last_fullsized":"2023-09-30T00:17:51.585090+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971832+0000","last_change":"2023-09-30T00:17:51.957745+0000","last_active":"2023-09-30T00:17:51.971832+0000","last_peered":"2023-09-30T00:17:51.971832+0000","last_clean":"2023-09-30T00:17:51.971832+0000","last_became_active":"2023-09-30T00:17:51.955331+0000","last_became_peered":"2023-09-30T00:17:51.955331+0000","last_unstale":"2023-09-30T00:17:51.971832+0000","last_undegraded":"2023-09-30T00:17:51.971832+0000","last_fullsized":"2023-09-30T00:17:51.971832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:45.393660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973877+0000","last_change":"2023-09-30T00:17:51.596021+0000","last_active":"2023-09-30T00:17:51.973877+0000","last_peered":"2023-09-30T00:17:51.973877+0000","last_clean":"2023-09-30T00:17:51.973877+0000","last_became_active":"2023-09-30T00:17:51.595745+0000","last_became_peered":"2023-09-30T00:17:51.595745+0000","last_unstale":"2023-09-30T00:17:51.973877+0000","last_undegraded":"2023-09-30T00:17:51.973877+0000","last_fullsized":"2023-09-30T00:17:51.973877+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:44.390191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.569794+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:06:14.351937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974567+0000","last_change":"2023-09-30T00:17:51.587276+0000","last_active":"2023-09-30T00:17:51.974567+0000","last_peered":"2023-09-30T00:17:51.974567+0000","last_clean":"2023-09-30T00:17:51.974567+0000","last_became_active":"2023-09-30T00:17:51.586984+0000","last_became_peered":"2023-09-30T00:17:51.586984+0000","last_unstale":"2023-09-30T00:17:51.974567+0000","last_undegraded":"2023-09-30T00:17:51.974567+0000","last_fullsized":"2023-09-30T00:17:51.974567+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.953787+0000","last_change":"2023-09-30T00:17:51.953787+0000","last_active":"2023-09-30T00:17:51.953787+0000","last_peered":"2023-09-30T00:17:51.953787+0000","last_clean":"2023-09-30T00:17:51.953787+0000","last_became_active":"2023-09-30T00:17:51.951805+0000","last_became_peered":"2023-09-30T00:17:51.951805+0000","last_unstale":"2023-09-30T00:17:51.953787+0000","last_undegraded":"2023-09-30T00:17:51.953787+0000","last_fullsized":"2023-09-30T00:17:51.953787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:57:04.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965610+0000","last_change":"2023-09-30T00:17:51.593225+0000","last_active":"2023-09-30T00:17:51.965610+0000","last_peered":"2023-09-30T00:17:51.965610+0000","last_clean":"2023-09-30T00:17:51.965610+0000","last_became_active":"2023-09-30T00:17:51.592913+0000","last_became_peered":"2023-09-30T00:17:51.592913+0000","last_unstale":"2023-09-30T00:17:51.965610+0000","last_undegraded":"2023-09-30T00:17:51.965610+0000","last_fullsized":"2023-09-30T00:17:51.965610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:37:56.776343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.955058+0000","last_change":"2023-09-30T00:17:51.955058+0000","last_active":"2023-09-30T00:17:51.955058+0000","last_peered":"2023-09-30T00:17:51.955058+0000","last_clean":"2023-09-30T00:17:51.955058+0000","last_became_active":"2023-09-30T00:17:51.954787+0000","last_became_peered":"2023-09-30T00:17:51.954787+0000","last_unstale":"2023-09-30T00:17:51.955058+0000","last_undegraded":"2023-09-30T00:17:51.955058+0000","last_fullsized":"2023-09-30T00:17:51.955058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:49:30.148218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":8,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.586229+0000","last_change":"2023-09-30T00:17:51.586229+0000","last_active":"2023-09-30T00:17:51.586229+0000","last_peered":"2023-09-30T00:17:51.586229+0000","last_clean":"2023-09-30T00:17:51.586229+0000","last_became_active":"2023-09-30T00:17:51.586027+0000","last_became_peered":"2023-09-30T00:17:51.586027+0000","last_unstale":"2023-09-30T00:17:51.586229+0000","last_undegraded":"2023-09-30T00:17:51.586229+0000","last_fullsized":"2023-09-30T00:17:51.586229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:48:13.817514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962963+0000","last_change":"2023-09-30T00:17:51.959628+0000","last_active":"2023-09-30T00:17:51.962963+0000","last_peered":"2023-09-30T00:17:51.962963+0000","last_clean":"2023-09-30T00:17:51.962963+0000","last_became_active":"2023-09-30T00:17:51.958974+0000","last_became_peered":"2023-09-30T00:17:51.958974+0000","last_unstale":"2023-09-30T00:17:51.962963+0000","last_undegraded":"2023-09-30T00:17:51.962963+0000","last_fullsized":"2023-09-30T00:17:51.962963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:28.604134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"18'4","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960480+0000","last_change":"2023-09-30T00:17:51.590387+0000","last_active":"2023-09-30T00:17:51.960480+0000","last_peered":"2023-09-30T00:17:51.960480+0000","last_clean":"2023-09-30T00:17:51.960480+0000","last_became_active":"2023-09-30T00:17:51.585479+0000","last_became_peered":"2023-09-30T00:17:51.585479+0000","last_unstale":"2023-09-30T00:17:51.960480+0000","last_undegraded":"2023-09-30T00:17:51.960480+0000","last_fullsized":"2023-09-30T00:17:51.960480+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":8,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.949418+0000","last_change":"2023-09-30T00:17:51.949418+0000","last_active":"2023-09-30T00:17:51.949418+0000","last_peered":"2023-09-30T00:17:51.949418+0000","last_clean":"2023-09-30T00:17:51.949418+0000","last_became_active":"2023-09-30T00:17:51.949101+0000","last_became_peered":"2023-09-30T00:17:51.949101+0000","last_unstale":"2023-09-30T00:17:51.949418+0000","last_undegraded":"2023-09-30T00:17:51.949418+0000","last_fullsized":"2023-09-30T00:17:51.949418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975274+0000","last_change":"2023-09-30T00:17:51.964155+0000","last_active":"2023-09-30T00:17:51.975274+0000","last_peered":"2023-09-30T00:17:51.975274+0000","last_clean":"2023-09-30T00:17:51.975274+0000","last_became_active":"2023-09-30T00:17:51.958334+0000","last_became_peered":"2023-09-30T00:17:51.958334+0000","last_unstale":"2023-09-30T00:17:51.975274+0000","last_undegraded":"2023-09-30T00:17:51.975274+0000","last_fullsized":"2023-09-30T00:17:51.975274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570057+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:31:59.960264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976104+0000","last_change":"2023-09-30T00:17:51.970791+0000","last_active":"2023-09-30T00:17:51.976104+0000","last_peered":"2023-09-30T00:17:51.976104+0000","last_clean":"2023-09-30T00:17:51.976104+0000","last_became_active":"2023-09-30T00:17:51.964142+0000","last_became_peered":"2023-09-30T00:17:51.964142+0000","last_unstale":"2023-09-30T00:17:51.976104+0000","last_undegraded":"2023-09-30T00:17:51.976104+0000","last_fullsized":"2023-09-30T00:17:51.976104+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.568144+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:18:28.873560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-09-30T00:17:51.564162+0000","last_change":"2023-09-30T00:17:51.570896+0000","last_active":"2023-09-30T00:17:51.564162+0000","last_peered":"2023-09-30T00:17:51.564162+0000","last_clean":"2023-09-30T00:17:51.564162+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-30T00:17:51.564162+0000","last_undegraded":"2023-09-30T00:17:51.564162+0000","last_fullsized":"2023-09-30T00:17:51.564162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:02:47.475511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672965,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9220,"kb_used_data":764,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362620,"statfs":{"total":96636764160,"available":96627322880,"internally_reserved":0,"allocated":782336,"data_stored":646622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]}]},{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8060,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363780,"statfs":{"total":96636764160,"available":96628510720,"internally_reserved":0,"allocated":184320,"data_stored":55478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8852,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94362988,"statfs":{"total":96636764160,"available":96627699712,"internally_reserved":0,"allocated":733184,"data_stored":634680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8052,"kb_used_data":172,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363788,"statfs":{"total":96636764160,"available":96628518912,"internally_reserved":0,"allocated":176128,"data_stored":55170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.859}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7779999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:18:00.475 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:18:00.601 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.601+0000 7f9ad47f6700 1 Processor -- start 2023-09-30T00:18:00.602 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.601+0000 7f9ad47f6700 1 -- start start 2023-09-30T00:18:00.603 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad47f6700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9acc0a0dd0 0x7f9acc14c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:00.604 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad47f6700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc071020 con 0x7f9acc070c50 2023-09-30T00:18:00.604 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad47f6700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc0b76a0 con 0x7f9acc0b7160 2023-09-30T00:18:00.604 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad47f6700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9acc0b7940 con 0x7f9acc0a0dd0 2023-09-30T00:18:00.604 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad2d93700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9acc0a0dd0 0x7f9acc14c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:00.604 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad2592700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9acc070c50 0x7f9acc0a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50206/0 (socket says 172.21.15.94:50206) 2023-09-30T00:18:00.604 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad2592700 1 -- 172.21.15.94:0/544815440 learned_addr learned my addr 172.21.15.94:0/544815440 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:00.604 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad2d93700 1 -- 172.21.15.94:0/544815440 >> v1:172.21.15.121:6789/0 conn(0x7f9acc0b7160 legacy=0x7f9acc13fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:00.604 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad2d93700 1 -- 172.21.15.94:0/544815440 >> v1:172.21.15.94:6789/0 conn(0x7f9acc070c50 legacy=0x7f9acc0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:00.605 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad2d93700 1 -- 172.21.15.94:0/544815440 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc0a2070 con 0x7f9acc0a0dd0 2023-09-30T00:18:00.605 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad1590700 1 -- 172.21.15.94:0/544815440 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118477436 0 0) 0x7f9acc071020 con 0x7f9acc070c50 2023-09-30T00:18:00.605 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad2d93700 1 --2- 172.21.15.94:0/544815440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9acc0a0dd0 0x7f9acc14c0d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4009e60 tx=0x7f9ac4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=996a9300e20a6a33 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:00.605 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad1590700 1 -- 172.21.15.94:0/544815440 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac400e070 con 0x7f9acc0a0dd0 2023-09-30T00:18:00.605 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad1590700 1 -- 172.21.15.94:0/544815440 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac400bcc0 con 0x7f9acc0a0dd0 2023-09-30T00:18:00.605 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.602+0000 7f9ad1590700 1 -- 172.21.15.94:0/544815440 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4012680 con 0x7f9acc0a0dd0 2023-09-30T00:18:00.605 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.603+0000 7f9ad47f6700 1 -- 172.21.15.94:0/544815440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9acc0a0dd0 msgr2=0x7f9acc14c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:00.605 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.603+0000 7f9ad47f6700 1 --2- 172.21.15.94:0/544815440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9acc0a0dd0 0x7f9acc14c0d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4009e60 tx=0x7f9ac4009780 comp rx=0 tx=0).stop 2023-09-30T00:18:00.605 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.603+0000 7f9ad47f6700 1 -- 172.21.15.94:0/544815440 shutdown_connections 2023-09-30T00:18:00.606 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.603+0000 7f9ad47f6700 1 --2- 172.21.15.94:0/544815440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9acc0a0dd0 0x7f9acc14c0d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.606 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.603+0000 7f9ad47f6700 1 -- 172.21.15.94:0/544815440 >> 172.21.15.94:0/544815440 conn(0x7f9acc06a090 msgr2=0x7f9acc0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:00.606 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.603+0000 7f9ad47f6700 1 -- 172.21.15.94:0/544815440 shutdown_connections 2023-09-30T00:18:00.606 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.603+0000 7f9ad47f6700 1 -- 172.21.15.94:0/544815440 wait complete. 2023-09-30T00:18:00.608 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.604+0000 7f9ad47f6700 1 Processor -- start 2023-09-30T00:18:00.609 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.604+0000 7f9ad47f6700 1 -- start start 2023-09-30T00:18:00.609 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad47f6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9acc070c50 0x7f9acc13ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:00.609 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad47f6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9acc0a0dd0 0x7f9acc0a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:00.609 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad47f6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9acc0b7160 0x7f9acc14fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:00.609 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad47f6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc13a6e0 con 0x7f9acc0b7160 2023-09-30T00:18:00.609 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad47f6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc1399d0 con 0x7f9acc070c50 2023-09-30T00:18:00.609 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad47f6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc154210 con 0x7f9acc0a0dd0 2023-09-30T00:18:00.609 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad2d93700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9acc0b7160 0x7f9acc14fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:00.609 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad2d93700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9acc0b7160 0x7f9acc14fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50594/0 (socket says 172.21.15.94:50594) 2023-09-30T00:18:00.610 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad2d93700 1 -- 172.21.15.94:0/1217232712 learned_addr learned my addr 172.21.15.94:0/1217232712 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:00.610 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad1d91700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9acc0a0dd0 0x7f9acc0a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:00.610 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad2592700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9acc070c50 0x7f9acc13ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:00.610 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad2d93700 1 -- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9acc0a0dd0 msgr2=0x7f9acc0a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:00.610 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad2d93700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9acc0a0dd0 0x7f9acc0a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.610 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad2d93700 1 -- 172.21.15.94:0/1217232712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9acc070c50 msgr2=0x7f9acc13ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:00.610 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad2d93700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9acc070c50 0x7f9acc13ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.610 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad2d93700 1 -- 172.21.15.94:0/1217232712 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac4009a20 con 0x7f9acc0b7160 2023-09-30T00:18:00.610 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad1d91700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9acc0a0dd0 0x7f9acc0a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:00.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad2d93700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9acc0b7160 0x7f9acc14fc00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9acc0a97e0 tx=0x7f9ac4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:00.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ac27fc700 1 -- 172.21.15.94:0/1217232712 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac400beb0 con 0x7f9acc0b7160 2023-09-30T00:18:00.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ac27fc700 1 -- 172.21.15.94:0/1217232712 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac401b630 con 0x7f9acc0b7160 2023-09-30T00:18:00.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad47f6700 1 -- 172.21.15.94:0/1217232712 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9acc1544d0 con 0x7f9acc0b7160 2023-09-30T00:18:00.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ac27fc700 1 -- 172.21.15.94:0/1217232712 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac401ba10 con 0x7f9acc0b7160 2023-09-30T00:18:00.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.605+0000 7f9ad47f6700 1 -- 172.21.15.94:0/1217232712 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9acc13c390 con 0x7f9acc0b7160 2023-09-30T00:18:00.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.606+0000 7f9ad47f6700 1 -- 172.21.15.94:0/1217232712 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9acc13c580 con 0x7f9acc0b7160 2023-09-30T00:18:00.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.607+0000 7f9ac27fc700 1 -- 172.21.15.94:0/1217232712 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f9ac4003020 con 0x7f9acc0b7160 2023-09-30T00:18:00.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.607+0000 7f9ac27fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:00.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.608+0000 7f9ac27fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:00.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.608+0000 7f9ac27fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:00.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.608+0000 7f9ac27fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:00.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.608+0000 7f9ac27fc700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9ab00987b0 0x7f9ab009ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:00.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.608+0000 7f9ad2592700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9ab00987b0 0x7f9ab009ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:00.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.608+0000 7f9ac27fc700 1 -- 172.21.15.94:0/1217232712 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f9ac40c2280 con 0x7f9acc0b7160 2023-09-30T00:18:00.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.608+0000 7f9ad2592700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9ab00987b0 0x7f9ab009ac60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9abc007fd0 tx=0x7f9abc007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:00.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.611+0000 7f9ac27fc700 1 -- 172.21.15.94:0/1217232712 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ac408f9c0 con 0x7f9acc0b7160 2023-09-30T00:18:00.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.788+0000 7f9ad47f6700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-30T00:18:00.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.788+0000 7f9ad47f6700 1 -- 172.21.15.94:0/1217232712 --> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9acc13b0b0 con 0x7f9ab00987b0 2023-09-30T00:18:00.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.795+0000 7f9ac27fc700 1 -- 172.21.15.94:0/1217232712 <== mgr.4106 v2:172.21.15.94:6808/84242 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+269336 (secure 0 0 0) 0x7f9acc13b0b0 con 0x7f9ab00987b0 2023-09-30T00:18:00.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.795+0000 7f9ac27fc700 20 mgrc handle_command_reply tid 0 r 0 2023-09-30T00:18:00.798 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:00.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 10 mgrc shutdown 2023-09-30T00:18:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 1 -- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9ab00987b0 msgr2=0x7f9ab009ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9ab00987b0 0x7f9ab009ac60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9abc007fd0 tx=0x7f9abc007d30 comp rx=0 tx=0).stop 2023-09-30T00:18:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 1 -- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9acc0b7160 msgr2=0x7f9acc14fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9acc0b7160 0x7f9acc14fc00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9acc0a97e0 tx=0x7f9ac4019000 comp rx=0 tx=0).stop 2023-09-30T00:18:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 1 -- 172.21.15.94:0/1217232712 shutdown_connections 2023-09-30T00:18:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9ab00987b0 0x7f9ab009ac60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9acc0b7160 0x7f9acc14fc00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9acc070c50 0x7f9acc13ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 1 --2- 172.21.15.94:0/1217232712 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9acc0a0dd0 0x7f9acc0a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 1 -- 172.21.15.94:0/1217232712 >> 172.21.15.94:0/1217232712 conn(0x7f9acc06a090 msgr2=0x7f9acc13fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:00.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 1 -- 172.21.15.94:0/1217232712 shutdown_connections 2023-09-30T00:18:00.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.798+0000 7f9ad47f6700 1 -- 172.21.15.94:0/1217232712 wait complete. 2023-09-30T00:18:00.802 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2023-09-30T00:18:00.810 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-09-30T00:17:59.852121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":34452,"kb_used_data":1908,"kb_used_omap":0,"kb_used_meta":32512,"kb_avail":377452908,"statfs":{"total":386547056640,"available":386511777792,"internally_reserved":0,"allocated":1953792,"data_stored":1422368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33292288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001648"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962397+0000","last_change":"2023-09-30T00:17:51.960193+0000","last_active":"2023-09-30T00:17:51.962397+0000","last_peered":"2023-09-30T00:17:51.962397+0000","last_clean":"2023-09-30T00:17:51.962397+0000","last_became_active":"2023-09-30T00:17:51.959888+0000","last_became_peered":"2023-09-30T00:17:51.959888+0000","last_unstale":"2023-09-30T00:17:51.962397+0000","last_undegraded":"2023-09-30T00:17:51.962397+0000","last_fullsized":"2023-09-30T00:17:51.962397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:40:42.458928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972435+0000","last_change":"2023-09-30T00:17:51.968577+0000","last_active":"2023-09-30T00:17:51.972435+0000","last_peered":"2023-09-30T00:17:51.972435+0000","last_clean":"2023-09-30T00:17:51.972435+0000","last_became_active":"2023-09-30T00:17:51.968275+0000","last_became_peered":"2023-09-30T00:17:51.968275+0000","last_unstale":"2023-09-30T00:17:51.972435+0000","last_undegraded":"2023-09-30T00:17:51.972435+0000","last_fullsized":"2023-09-30T00:17:51.972435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:19:24.974156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977703+0000","last_change":"2023-09-30T00:17:51.974359+0000","last_active":"2023-09-30T00:17:51.977703+0000","last_peered":"2023-09-30T00:17:51.977703+0000","last_clean":"2023-09-30T00:17:51.977703+0000","last_became_active":"2023-09-30T00:17:51.973789+0000","last_became_peered":"2023-09-30T00:17:51.973789+0000","last_unstale":"2023-09-30T00:17:51.977703+0000","last_undegraded":"2023-09-30T00:17:51.977703+0000","last_fullsized":"2023-09-30T00:17:51.977703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:27.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971859+0000","last_change":"2023-09-30T00:17:51.958127+0000","last_active":"2023-09-30T00:17:51.971859+0000","last_peered":"2023-09-30T00:17:51.971859+0000","last_clean":"2023-09-30T00:17:51.971859+0000","last_became_active":"2023-09-30T00:17:51.957850+0000","last_became_peered":"2023-09-30T00:17:51.957850+0000","last_unstale":"2023-09-30T00:17:51.971859+0000","last_undegraded":"2023-09-30T00:17:51.971859+0000","last_fullsized":"2023-09-30T00:17:51.971859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:48:36.950659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973474+0000","last_change":"2023-09-30T00:17:51.966792+0000","last_active":"2023-09-30T00:17:51.973474+0000","last_peered":"2023-09-30T00:17:51.973474+0000","last_clean":"2023-09-30T00:17:51.973474+0000","last_became_active":"2023-09-30T00:17:51.965887+0000","last_became_peered":"2023-09-30T00:17:51.965887+0000","last_unstale":"2023-09-30T00:17:51.973474+0000","last_undegraded":"2023-09-30T00:17:51.973474+0000","last_fullsized":"2023-09-30T00:17:51.973474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973636+0000","last_change":"2023-09-30T00:17:51.967175+0000","last_active":"2023-09-30T00:17:51.973636+0000","last_peered":"2023-09-30T00:17:51.973636+0000","last_clean":"2023-09-30T00:17:51.973636+0000","last_became_active":"2023-09-30T00:17:51.966408+0000","last_became_peered":"2023-09-30T00:17:51.966408+0000","last_unstale":"2023-09-30T00:17:51.973636+0000","last_undegraded":"2023-09-30T00:17:51.973636+0000","last_fullsized":"2023-09-30T00:17:51.973636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:47:56.029102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972783+0000","last_change":"2023-09-30T00:17:51.964264+0000","last_active":"2023-09-30T00:17:51.972783+0000","last_peered":"2023-09-30T00:17:51.972783+0000","last_clean":"2023-09-30T00:17:51.972783+0000","last_became_active":"2023-09-30T00:17:51.962740+0000","last_became_peered":"2023-09-30T00:17:51.962740+0000","last_unstale":"2023-09-30T00:17:51.972783+0000","last_undegraded":"2023-09-30T00:17:51.972783+0000","last_fullsized":"2023-09-30T00:17:51.972783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:52:26.654463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975296+0000","last_change":"2023-09-30T00:17:51.967599+0000","last_active":"2023-09-30T00:17:51.975296+0000","last_peered":"2023-09-30T00:17:51.975296+0000","last_clean":"2023-09-30T00:17:51.975296+0000","last_became_active":"2023-09-30T00:17:51.961762+0000","last_became_peered":"2023-09-30T00:17:51.961762+0000","last_unstale":"2023-09-30T00:17:51.975296+0000","last_undegraded":"2023-09-30T00:17:51.975296+0000","last_fullsized":"2023-09-30T00:17:51.975296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:35:31.620523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971780+0000","last_change":"2023-09-30T00:17:51.971131+0000","last_active":"2023-09-30T00:17:51.971780+0000","last_peered":"2023-09-30T00:17:51.971780+0000","last_clean":"2023-09-30T00:17:51.971780+0000","last_became_active":"2023-09-30T00:17:51.970865+0000","last_became_peered":"2023-09-30T00:17:51.970865+0000","last_unstale":"2023-09-30T00:17:51.971780+0000","last_undegraded":"2023-09-30T00:17:51.971780+0000","last_fullsized":"2023-09-30T00:17:51.971780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:53:33.380570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972400+0000","last_change":"2023-09-30T00:17:51.963624+0000","last_active":"2023-09-30T00:17:51.972400+0000","last_peered":"2023-09-30T00:17:51.972400+0000","last_clean":"2023-09-30T00:17:51.972400+0000","last_became_active":"2023-09-30T00:17:51.963303+0000","last_became_peered":"2023-09-30T00:17:51.963303+0000","last_unstale":"2023-09-30T00:17:51.972400+0000","last_undegraded":"2023-09-30T00:17:51.972400+0000","last_fullsized":"2023-09-30T00:17:51.972400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971954+0000","last_change":"2023-09-30T00:17:51.961620+0000","last_active":"2023-09-30T00:17:51.971954+0000","last_peered":"2023-09-30T00:17:51.971954+0000","last_clean":"2023-09-30T00:17:51.971954+0000","last_became_active":"2023-09-30T00:17:51.958237+0000","last_became_peered":"2023-09-30T00:17:51.958237+0000","last_unstale":"2023-09-30T00:17:51.971954+0000","last_undegraded":"2023-09-30T00:17:51.971954+0000","last_fullsized":"2023-09-30T00:17:51.971954+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977575+0000","last_change":"2023-09-30T00:17:51.974397+0000","last_active":"2023-09-30T00:17:51.977575+0000","last_peered":"2023-09-30T00:17:51.977575+0000","last_clean":"2023-09-30T00:17:51.977575+0000","last_became_active":"2023-09-30T00:17:51.974025+0000","last_became_peered":"2023-09-30T00:17:51.974025+0000","last_unstale":"2023-09-30T00:17:51.977575+0000","last_undegraded":"2023-09-30T00:17:51.977575+0000","last_fullsized":"2023-09-30T00:17:51.977575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:49:07.723751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.966627+0000","last_change":"2023-09-30T00:17:51.966187+0000","last_active":"2023-09-30T00:17:51.966627+0000","last_peered":"2023-09-30T00:17:51.966627+0000","last_clean":"2023-09-30T00:17:51.966627+0000","last_became_active":"2023-09-30T00:17:51.965900+0000","last_became_peered":"2023-09-30T00:17:51.965900+0000","last_unstale":"2023-09-30T00:17:51.966627+0000","last_undegraded":"2023-09-30T00:17:51.966627+0000","last_fullsized":"2023-09-30T00:17:51.966627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:30:07.208946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976147+0000","last_change":"2023-09-30T00:17:51.970985+0000","last_active":"2023-09-30T00:17:51.976147+0000","last_peered":"2023-09-30T00:17:51.976147+0000","last_clean":"2023-09-30T00:17:51.976147+0000","last_became_active":"2023-09-30T00:17:51.966979+0000","last_became_peered":"2023-09-30T00:17:51.966979+0000","last_unstale":"2023-09-30T00:17:51.976147+0000","last_undegraded":"2023-09-30T00:17:51.976147+0000","last_fullsized":"2023-09-30T00:17:51.976147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:09:28.833739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975911+0000","last_change":"2023-09-30T00:17:51.970398+0000","last_active":"2023-09-30T00:17:51.975911+0000","last_peered":"2023-09-30T00:17:51.975911+0000","last_clean":"2023-09-30T00:17:51.975911+0000","last_became_active":"2023-09-30T00:17:51.961675+0000","last_became_peered":"2023-09-30T00:17:51.961675+0000","last_unstale":"2023-09-30T00:17:51.975911+0000","last_undegraded":"2023-09-30T00:17:51.975911+0000","last_fullsized":"2023-09-30T00:17:51.975911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:33:21.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972718+0000","last_change":"2023-09-30T00:17:51.969081+0000","last_active":"2023-09-30T00:17:51.972718+0000","last_peered":"2023-09-30T00:17:51.972718+0000","last_clean":"2023-09-30T00:17:51.972718+0000","last_became_active":"2023-09-30T00:17:51.966447+0000","last_became_peered":"2023-09-30T00:17:51.966447+0000","last_unstale":"2023-09-30T00:17:51.972718+0000","last_undegraded":"2023-09-30T00:17:51.972718+0000","last_fullsized":"2023-09-30T00:17:51.972718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:19:12.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960350+0000","last_change":"2023-09-30T00:17:51.953980+0000","last_active":"2023-09-30T00:17:51.960350+0000","last_peered":"2023-09-30T00:17:51.960350+0000","last_clean":"2023-09-30T00:17:51.960350+0000","last_became_active":"2023-09-30T00:17:51.952191+0000","last_became_peered":"2023-09-30T00:17:51.952191+0000","last_unstale":"2023-09-30T00:17:51.960350+0000","last_undegraded":"2023-09-30T00:17:51.960350+0000","last_fullsized":"2023-09-30T00:17:51.960350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:52:46.774408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973554+0000","last_change":"2023-09-30T00:17:51.969589+0000","last_active":"2023-09-30T00:17:51.973554+0000","last_peered":"2023-09-30T00:17:51.973554+0000","last_clean":"2023-09-30T00:17:51.973554+0000","last_became_active":"2023-09-30T00:17:51.967300+0000","last_became_peered":"2023-09-30T00:17:51.967300+0000","last_unstale":"2023-09-30T00:17:51.973554+0000","last_undegraded":"2023-09-30T00:17:51.973554+0000","last_fullsized":"2023-09-30T00:17:51.973554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:22:57.623135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.966670+0000","last_change":"2023-09-30T00:17:51.966047+0000","last_active":"2023-09-30T00:17:51.966670+0000","last_peered":"2023-09-30T00:17:51.966670+0000","last_clean":"2023-09-30T00:17:51.966670+0000","last_became_active":"2023-09-30T00:17:51.965457+0000","last_became_peered":"2023-09-30T00:17:51.965457+0000","last_unstale":"2023-09-30T00:17:51.966670+0000","last_undegraded":"2023-09-30T00:17:51.966670+0000","last_fullsized":"2023-09-30T00:17:51.966670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:24:33.105806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972047+0000","last_change":"2023-09-30T00:17:51.961502+0000","last_active":"2023-09-30T00:17:51.972047+0000","last_peered":"2023-09-30T00:17:51.972047+0000","last_clean":"2023-09-30T00:17:51.972047+0000","last_became_active":"2023-09-30T00:17:51.961191+0000","last_became_peered":"2023-09-30T00:17:51.961191+0000","last_unstale":"2023-09-30T00:17:51.972047+0000","last_undegraded":"2023-09-30T00:17:51.972047+0000","last_fullsized":"2023-09-30T00:17:51.972047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:05:37.196886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976121+0000","last_change":"2023-09-30T00:17:51.970789+0000","last_active":"2023-09-30T00:17:51.976121+0000","last_peered":"2023-09-30T00:17:51.976121+0000","last_clean":"2023-09-30T00:17:51.976121+0000","last_became_active":"2023-09-30T00:17:51.967616+0000","last_became_peered":"2023-09-30T00:17:51.967616+0000","last_unstale":"2023-09-30T00:17:51.976121+0000","last_undegraded":"2023-09-30T00:17:51.976121+0000","last_fullsized":"2023-09-30T00:17:51.976121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976127+0000","last_change":"2023-09-30T00:17:51.970797+0000","last_active":"2023-09-30T00:17:51.976127+0000","last_peered":"2023-09-30T00:17:51.976127+0000","last_clean":"2023-09-30T00:17:51.976127+0000","last_became_active":"2023-09-30T00:17:51.963382+0000","last_became_peered":"2023-09-30T00:17:51.963382+0000","last_unstale":"2023-09-30T00:17:51.976127+0000","last_undegraded":"2023-09-30T00:17:51.976127+0000","last_fullsized":"2023-09-30T00:17:51.976127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:35:22.077360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972764+0000","last_change":"2023-09-30T00:17:51.963058+0000","last_active":"2023-09-30T00:17:51.972764+0000","last_peered":"2023-09-30T00:17:51.972764+0000","last_clean":"2023-09-30T00:17:51.972764+0000","last_became_active":"2023-09-30T00:17:51.959450+0000","last_became_peered":"2023-09-30T00:17:51.959450+0000","last_unstale":"2023-09-30T00:17:51.972764+0000","last_undegraded":"2023-09-30T00:17:51.972764+0000","last_fullsized":"2023-09-30T00:17:51.972764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:41:36.585190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975442+0000","last_change":"2023-09-30T00:17:51.967581+0000","last_active":"2023-09-30T00:17:51.975442+0000","last_peered":"2023-09-30T00:17:51.975442+0000","last_clean":"2023-09-30T00:17:51.975442+0000","last_became_active":"2023-09-30T00:17:51.961348+0000","last_became_peered":"2023-09-30T00:17:51.961348+0000","last_unstale":"2023-09-30T00:17:51.975442+0000","last_undegraded":"2023-09-30T00:17:51.975442+0000","last_fullsized":"2023-09-30T00:17:51.975442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:11:09.140469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975871+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.975871+0000","last_peered":"2023-09-30T00:17:51.975871+0000","last_clean":"2023-09-30T00:17:51.975871+0000","last_became_active":"2023-09-30T00:17:51.963534+0000","last_became_peered":"2023-09-30T00:17:51.963534+0000","last_unstale":"2023-09-30T00:17:51.975871+0000","last_undegraded":"2023-09-30T00:17:51.975871+0000","last_fullsized":"2023-09-30T00:17:51.975871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:21:49.086269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.961574+0000","last_change":"2023-09-30T00:17:51.961210+0000","last_active":"2023-09-30T00:17:51.961574+0000","last_peered":"2023-09-30T00:17:51.961574+0000","last_clean":"2023-09-30T00:17:51.961574+0000","last_became_active":"2023-09-30T00:17:51.961010+0000","last_became_peered":"2023-09-30T00:17:51.961010+0000","last_unstale":"2023-09-30T00:17:51.961574+0000","last_undegraded":"2023-09-30T00:17:51.961574+0000","last_fullsized":"2023-09-30T00:17:51.961574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:31:50.854220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972810+0000","last_change":"2023-09-30T00:17:51.968803+0000","last_active":"2023-09-30T00:17:51.972810+0000","last_peered":"2023-09-30T00:17:51.972810+0000","last_clean":"2023-09-30T00:17:51.972810+0000","last_became_active":"2023-09-30T00:17:51.965833+0000","last_became_peered":"2023-09-30T00:17:51.965833+0000","last_unstale":"2023-09-30T00:17:51.972810+0000","last_undegraded":"2023-09-30T00:17:51.972810+0000","last_fullsized":"2023-09-30T00:17:51.972810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960606+0000","last_change":"2023-09-30T00:17:51.592497+0000","last_active":"2023-09-30T00:17:51.960606+0000","last_peered":"2023-09-30T00:17:51.960606+0000","last_clean":"2023-09-30T00:17:51.960606+0000","last_became_active":"2023-09-30T00:17:51.585128+0000","last_became_peered":"2023-09-30T00:17:51.585128+0000","last_unstale":"2023-09-30T00:17:51.960606+0000","last_undegraded":"2023-09-30T00:17:51.960606+0000","last_fullsized":"2023-09-30T00:17:51.960606+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:46.058014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977744+0000","last_change":"2023-09-30T00:17:51.974520+0000","last_active":"2023-09-30T00:17:51.977744+0000","last_peered":"2023-09-30T00:17:51.977744+0000","last_clean":"2023-09-30T00:17:51.977744+0000","last_became_active":"2023-09-30T00:17:51.974109+0000","last_became_peered":"2023-09-30T00:17:51.974109+0000","last_unstale":"2023-09-30T00:17:51.977744+0000","last_undegraded":"2023-09-30T00:17:51.977744+0000","last_fullsized":"2023-09-30T00:17:51.977744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:25:05.134336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941011+0000","last_change":"2023-09-30T00:17:51.587986+0000","last_active":"2023-09-30T00:17:51.941011+0000","last_peered":"2023-09-30T00:17:51.941011+0000","last_clean":"2023-09-30T00:17:51.941011+0000","last_became_active":"2023-09-30T00:17:51.587724+0000","last_became_peered":"2023-09-30T00:17:51.587724+0000","last_unstale":"2023-09-30T00:17:51.941011+0000","last_undegraded":"2023-09-30T00:17:51.941011+0000","last_fullsized":"2023-09-30T00:17:51.941011+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971773+0000","last_change":"2023-09-30T00:17:51.971105+0000","last_active":"2023-09-30T00:17:51.971773+0000","last_peered":"2023-09-30T00:17:51.971773+0000","last_clean":"2023-09-30T00:17:51.971773+0000","last_became_active":"2023-09-30T00:17:51.970846+0000","last_became_peered":"2023-09-30T00:17:51.970846+0000","last_unstale":"2023-09-30T00:17:51.971773+0000","last_undegraded":"2023-09-30T00:17:51.971773+0000","last_fullsized":"2023-09-30T00:17:51.971773+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:51.108522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952700+0000","last_change":"2023-09-30T00:17:51.596511+0000","last_active":"2023-09-30T00:17:51.952700+0000","last_peered":"2023-09-30T00:17:51.952700+0000","last_clean":"2023-09-30T00:17:51.952700+0000","last_became_active":"2023-09-30T00:17:51.596034+0000","last_became_peered":"2023-09-30T00:17:51.596034+0000","last_unstale":"2023-09-30T00:17:51.952700+0000","last_undegraded":"2023-09-30T00:17:51.952700+0000","last_fullsized":"2023-09-30T00:17:51.952700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:44:29.988610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969658+0000","last_change":"2023-09-30T00:17:51.964766+0000","last_active":"2023-09-30T00:17:51.969658+0000","last_peered":"2023-09-30T00:17:51.969658+0000","last_clean":"2023-09-30T00:17:51.969658+0000","last_became_active":"2023-09-30T00:17:51.964496+0000","last_became_peered":"2023-09-30T00:17:51.964496+0000","last_unstale":"2023-09-30T00:17:51.969658+0000","last_undegraded":"2023-09-30T00:17:51.969658+0000","last_fullsized":"2023-09-30T00:17:51.969658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.942081+0000","last_change":"2023-09-30T00:17:51.577022+0000","last_active":"2023-09-30T00:17:51.942081+0000","last_peered":"2023-09-30T00:17:51.942081+0000","last_clean":"2023-09-30T00:17:51.942081+0000","last_became_active":"2023-09-30T00:17:51.576726+0000","last_became_peered":"2023-09-30T00:17:51.576726+0000","last_unstale":"2023-09-30T00:17:51.942081+0000","last_undegraded":"2023-09-30T00:17:51.942081+0000","last_fullsized":"2023-09-30T00:17:51.942081+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972739+0000","last_change":"2023-09-30T00:17:51.969564+0000","last_active":"2023-09-30T00:17:51.972739+0000","last_peered":"2023-09-30T00:17:51.972739+0000","last_clean":"2023-09-30T00:17:51.972739+0000","last_became_active":"2023-09-30T00:17:51.969240+0000","last_became_peered":"2023-09-30T00:17:51.969240+0000","last_unstale":"2023-09-30T00:17:51.972739+0000","last_undegraded":"2023-09-30T00:17:51.972739+0000","last_fullsized":"2023-09-30T00:17:51.972739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:42.701890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.949738+0000","last_change":"2023-09-30T00:17:51.588716+0000","last_active":"2023-09-30T00:17:51.949738+0000","last_peered":"2023-09-30T00:17:51.949738+0000","last_clean":"2023-09-30T00:17:51.949738+0000","last_became_active":"2023-09-30T00:17:51.588454+0000","last_became_peered":"2023-09-30T00:17:51.588454+0000","last_unstale":"2023-09-30T00:17:51.949738+0000","last_undegraded":"2023-09-30T00:17:51.949738+0000","last_fullsized":"2023-09-30T00:17:51.949738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.958204+0000","last_change":"2023-09-30T00:17:51.957598+0000","last_active":"2023-09-30T00:17:51.958204+0000","last_peered":"2023-09-30T00:17:51.958204+0000","last_clean":"2023-09-30T00:17:51.958204+0000","last_became_active":"2023-09-30T00:17:51.957336+0000","last_became_peered":"2023-09-30T00:17:51.957336+0000","last_unstale":"2023-09-30T00:17:51.958204+0000","last_undegraded":"2023-09-30T00:17:51.958204+0000","last_fullsized":"2023-09-30T00:17:51.958204+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:02:32.124291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"18'3","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973987+0000","last_change":"2023-09-30T00:17:51.589988+0000","last_active":"2023-09-30T00:17:51.973987+0000","last_peered":"2023-09-30T00:17:51.973987+0000","last_clean":"2023-09-30T00:17:51.973987+0000","last_became_active":"2023-09-30T00:17:51.589729+0000","last_became_peered":"2023-09-30T00:17:51.589729+0000","last_unstale":"2023-09-30T00:17:51.973987+0000","last_undegraded":"2023-09-30T00:17:51.973987+0000","last_fullsized":"2023-09-30T00:17:51.973987+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975239+0000","last_change":"2023-09-30T00:17:51.967245+0000","last_active":"2023-09-30T00:17:51.975239+0000","last_peered":"2023-09-30T00:17:51.975239+0000","last_clean":"2023-09-30T00:17:51.975239+0000","last_became_active":"2023-09-30T00:17:51.963755+0000","last_became_peered":"2023-09-30T00:17:51.963755+0000","last_unstale":"2023-09-30T00:17:51.975239+0000","last_undegraded":"2023-09-30T00:17:51.975239+0000","last_fullsized":"2023-09-30T00:17:51.975239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:57:31.735059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"18'6","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.983073+0000","last_change":"2023-09-30T00:17:51.592307+0000","last_active":"2023-09-30T00:17:51.983073+0000","last_peered":"2023-09-30T00:17:51.983073+0000","last_clean":"2023-09-30T00:17:51.983073+0000","last_became_active":"2023-09-30T00:17:51.591753+0000","last_became_peered":"2023-09-30T00:17:51.591753+0000","last_unstale":"2023-09-30T00:17:51.983073+0000","last_undegraded":"2023-09-30T00:17:51.983073+0000","last_fullsized":"2023-09-30T00:17:51.983073+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:14:46.193926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968817+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.968817+0000","last_peered":"2023-09-30T00:17:51.968817+0000","last_clean":"2023-09-30T00:17:51.968817+0000","last_became_active":"2023-09-30T00:17:51.967118+0000","last_became_peered":"2023-09-30T00:17:51.967118+0000","last_unstale":"2023-09-30T00:17:51.968817+0000","last_undegraded":"2023-09-30T00:17:51.968817+0000","last_fullsized":"2023-09-30T00:17:51.968817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:50.694814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969722+0000","last_change":"2023-09-30T00:17:51.591516+0000","last_active":"2023-09-30T00:17:51.969722+0000","last_peered":"2023-09-30T00:17:51.969722+0000","last_clean":"2023-09-30T00:17:51.969722+0000","last_became_active":"2023-09-30T00:17:51.591206+0000","last_became_peered":"2023-09-30T00:17:51.591206+0000","last_unstale":"2023-09-30T00:17:51.969722+0000","last_undegraded":"2023-09-30T00:17:51.969722+0000","last_fullsized":"2023-09-30T00:17:51.969722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:22.807978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977754+0000","last_change":"2023-09-30T00:17:51.974628+0000","last_active":"2023-09-30T00:17:51.977754+0000","last_peered":"2023-09-30T00:17:51.977754+0000","last_clean":"2023-09-30T00:17:51.977754+0000","last_became_active":"2023-09-30T00:17:51.974276+0000","last_became_peered":"2023-09-30T00:17:51.974276+0000","last_unstale":"2023-09-30T00:17:51.977754+0000","last_undegraded":"2023-09-30T00:17:51.977754+0000","last_fullsized":"2023-09-30T00:17:51.977754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.946238+0000","last_change":"2023-09-30T00:17:51.589138+0000","last_active":"2023-09-30T00:17:51.946238+0000","last_peered":"2023-09-30T00:17:51.946238+0000","last_clean":"2023-09-30T00:17:51.946238+0000","last_became_active":"2023-09-30T00:17:51.588883+0000","last_became_peered":"2023-09-30T00:17:51.588883+0000","last_unstale":"2023-09-30T00:17:51.946238+0000","last_undegraded":"2023-09-30T00:17:51.946238+0000","last_fullsized":"2023-09-30T00:17:51.946238+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977625+0000","last_change":"2023-09-30T00:17:51.972690+0000","last_active":"2023-09-30T00:17:51.977625+0000","last_peered":"2023-09-30T00:17:51.977625+0000","last_clean":"2023-09-30T00:17:51.977625+0000","last_became_active":"2023-09-30T00:17:51.969872+0000","last_became_peered":"2023-09-30T00:17:51.969872+0000","last_unstale":"2023-09-30T00:17:51.977625+0000","last_undegraded":"2023-09-30T00:17:51.977625+0000","last_fullsized":"2023-09-30T00:17:51.977625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952325+0000","last_change":"2023-09-30T00:17:51.597410+0000","last_active":"2023-09-30T00:17:51.952325+0000","last_peered":"2023-09-30T00:17:51.952325+0000","last_clean":"2023-09-30T00:17:51.952325+0000","last_became_active":"2023-09-30T00:17:51.597168+0000","last_became_peered":"2023-09-30T00:17:51.597168+0000","last_unstale":"2023-09-30T00:17:51.952325+0000","last_undegraded":"2023-09-30T00:17:51.952325+0000","last_fullsized":"2023-09-30T00:17:51.952325+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:44:45.339994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973574+0000","last_change":"2023-09-30T00:17:51.969752+0000","last_active":"2023-09-30T00:17:51.973574+0000","last_peered":"2023-09-30T00:17:51.973574+0000","last_clean":"2023-09-30T00:17:51.973574+0000","last_became_active":"2023-09-30T00:17:51.968993+0000","last_became_peered":"2023-09-30T00:17:51.968993+0000","last_unstale":"2023-09-30T00:17:51.973574+0000","last_undegraded":"2023-09-30T00:17:51.973574+0000","last_fullsized":"2023-09-30T00:17:51.973574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"18'8","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974927+0000","last_change":"2023-09-30T00:17:51.592339+0000","last_active":"2023-09-30T00:17:51.974927+0000","last_peered":"2023-09-30T00:17:51.974927+0000","last_clean":"2023-09-30T00:17:51.974927+0000","last_became_active":"2023-09-30T00:17:51.592134+0000","last_became_peered":"2023-09-30T00:17:51.592134+0000","last_unstale":"2023-09-30T00:17:51.974927+0000","last_undegraded":"2023-09-30T00:17:51.974927+0000","last_fullsized":"2023-09-30T00:17:51.974927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.959394+0000","last_change":"2023-09-30T00:17:51.958723+0000","last_active":"2023-09-30T00:17:51.959394+0000","last_peered":"2023-09-30T00:17:51.959394+0000","last_clean":"2023-09-30T00:17:51.959394+0000","last_became_active":"2023-09-30T00:17:51.958465+0000","last_became_peered":"2023-09-30T00:17:51.958465+0000","last_unstale":"2023-09-30T00:17:51.959394+0000","last_undegraded":"2023-09-30T00:17:51.959394+0000","last_fullsized":"2023-09-30T00:17:51.959394+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:59:59.352777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968301+0000","last_change":"2023-09-30T00:17:51.577740+0000","last_active":"2023-09-30T00:17:51.968301+0000","last_peered":"2023-09-30T00:17:51.968301+0000","last_clean":"2023-09-30T00:17:51.968301+0000","last_became_active":"2023-09-30T00:17:51.577449+0000","last_became_peered":"2023-09-30T00:17:51.577449+0000","last_unstale":"2023-09-30T00:17:51.968301+0000","last_undegraded":"2023-09-30T00:17:51.968301+0000","last_fullsized":"2023-09-30T00:17:51.968301+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977756+0000","last_change":"2023-09-30T00:17:51.974425+0000","last_active":"2023-09-30T00:17:51.977756+0000","last_peered":"2023-09-30T00:17:51.977756+0000","last_clean":"2023-09-30T00:17:51.977756+0000","last_became_active":"2023-09-30T00:17:51.974143+0000","last_became_peered":"2023-09-30T00:17:51.974143+0000","last_unstale":"2023-09-30T00:17:51.977756+0000","last_undegraded":"2023-09-30T00:17:51.977756+0000","last_fullsized":"2023-09-30T00:17:51.977756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:04:54.968664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972091+0000","last_change":"2023-09-30T00:17:51.578710+0000","last_active":"2023-09-30T00:17:51.972091+0000","last_peered":"2023-09-30T00:17:51.972091+0000","last_clean":"2023-09-30T00:17:51.972091+0000","last_became_active":"2023-09-30T00:17:51.578450+0000","last_became_peered":"2023-09-30T00:17:51.578450+0000","last_unstale":"2023-09-30T00:17:51.972091+0000","last_undegraded":"2023-09-30T00:17:51.972091+0000","last_fullsized":"2023-09-30T00:17:51.972091+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.958310+0000","last_change":"2023-09-30T00:17:51.951367+0000","last_active":"2023-09-30T00:17:51.958310+0000","last_peered":"2023-09-30T00:17:51.958310+0000","last_clean":"2023-09-30T00:17:51.958310+0000","last_became_active":"2023-09-30T00:17:51.951031+0000","last_became_peered":"2023-09-30T00:17:51.951031+0000","last_unstale":"2023-09-30T00:17:51.958310+0000","last_undegraded":"2023-09-30T00:17:51.958310+0000","last_fullsized":"2023-09-30T00:17:51.958310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976462+0000","last_change":"2023-09-30T00:17:51.594234+0000","last_active":"2023-09-30T00:17:51.976462+0000","last_peered":"2023-09-30T00:17:51.976462+0000","last_clean":"2023-09-30T00:17:51.976462+0000","last_became_active":"2023-09-30T00:17:51.594049+0000","last_became_peered":"2023-09-30T00:17:51.594049+0000","last_unstale":"2023-09-30T00:17:51.976462+0000","last_undegraded":"2023-09-30T00:17:51.976462+0000","last_fullsized":"2023-09-30T00:17:51.976462+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:53:48.206055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968844+0000","last_change":"2023-09-30T00:17:51.965298+0000","last_active":"2023-09-30T00:17:51.968844+0000","last_peered":"2023-09-30T00:17:51.968844+0000","last_clean":"2023-09-30T00:17:51.968844+0000","last_became_active":"2023-09-30T00:17:51.964965+0000","last_became_peered":"2023-09-30T00:17:51.964965+0000","last_unstale":"2023-09-30T00:17:51.968844+0000","last_undegraded":"2023-09-30T00:17:51.968844+0000","last_fullsized":"2023-09-30T00:17:51.968844+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:33:20.583162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.955628+0000","last_change":"2023-09-30T00:17:51.592485+0000","last_active":"2023-09-30T00:17:51.955628+0000","last_peered":"2023-09-30T00:17:51.955628+0000","last_clean":"2023-09-30T00:17:51.955628+0000","last_became_active":"2023-09-30T00:17:51.592054+0000","last_became_peered":"2023-09-30T00:17:51.592054+0000","last_unstale":"2023-09-30T00:17:51.955628+0000","last_undegraded":"2023-09-30T00:17:51.955628+0000","last_fullsized":"2023-09-30T00:17:51.955628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:46:52.872573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.961561+0000","last_change":"2023-09-30T00:17:51.952057+0000","last_active":"2023-09-30T00:17:51.961561+0000","last_peered":"2023-09-30T00:17:51.961561+0000","last_clean":"2023-09-30T00:17:51.961561+0000","last_became_active":"2023-09-30T00:17:51.951537+0000","last_became_peered":"2023-09-30T00:17:51.951537+0000","last_unstale":"2023-09-30T00:17:51.961561+0000","last_undegraded":"2023-09-30T00:17:51.961561+0000","last_fullsized":"2023-09-30T00:17:51.961561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:20:25.199612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.966041+0000","last_change":"2023-09-30T00:17:51.578071+0000","last_active":"2023-09-30T00:17:51.966041+0000","last_peered":"2023-09-30T00:17:51.966041+0000","last_clean":"2023-09-30T00:17:51.966041+0000","last_became_active":"2023-09-30T00:17:51.577808+0000","last_became_peered":"2023-09-30T00:17:51.577808+0000","last_unstale":"2023-09-30T00:17:51.966041+0000","last_undegraded":"2023-09-30T00:17:51.966041+0000","last_fullsized":"2023-09-30T00:17:51.966041+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:20.626316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976183+0000","last_change":"2023-09-30T00:17:51.970822+0000","last_active":"2023-09-30T00:17:51.976183+0000","last_peered":"2023-09-30T00:17:51.976183+0000","last_clean":"2023-09-30T00:17:51.976183+0000","last_became_active":"2023-09-30T00:17:51.970520+0000","last_became_peered":"2023-09-30T00:17:51.970520+0000","last_unstale":"2023-09-30T00:17:51.976183+0000","last_undegraded":"2023-09-30T00:17:51.976183+0000","last_fullsized":"2023-09-30T00:17:51.976183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:07.347359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952250+0000","last_change":"2023-09-30T00:17:51.595723+0000","last_active":"2023-09-30T00:17:51.952250+0000","last_peered":"2023-09-30T00:17:51.952250+0000","last_clean":"2023-09-30T00:17:51.952250+0000","last_became_active":"2023-09-30T00:17:51.595473+0000","last_became_peered":"2023-09-30T00:17:51.595473+0000","last_unstale":"2023-09-30T00:17:51.952250+0000","last_undegraded":"2023-09-30T00:17:51.952250+0000","last_fullsized":"2023-09-30T00:17:51.952250+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:06.343890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980343+0000","last_change":"2023-09-30T00:17:51.977010+0000","last_active":"2023-09-30T00:17:51.980343+0000","last_peered":"2023-09-30T00:17:51.980343+0000","last_clean":"2023-09-30T00:17:51.980343+0000","last_became_active":"2023-09-30T00:17:51.976744+0000","last_became_peered":"2023-09-30T00:17:51.976744+0000","last_unstale":"2023-09-30T00:17:51.980343+0000","last_undegraded":"2023-09-30T00:17:51.980343+0000","last_fullsized":"2023-09-30T00:17:51.980343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:02:14.797729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952109+0000","last_change":"2023-09-30T00:17:51.595305+0000","last_active":"2023-09-30T00:17:51.952109+0000","last_peered":"2023-09-30T00:17:51.952109+0000","last_clean":"2023-09-30T00:17:51.952109+0000","last_became_active":"2023-09-30T00:17:51.595042+0000","last_became_peered":"2023-09-30T00:17:51.595042+0000","last_unstale":"2023-09-30T00:17:51.952109+0000","last_undegraded":"2023-09-30T00:17:51.952109+0000","last_fullsized":"2023-09-30T00:17:51.952109+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:54.681232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968754+0000","last_change":"2023-09-30T00:17:51.965816+0000","last_active":"2023-09-30T00:17:51.968754+0000","last_peered":"2023-09-30T00:17:51.968754+0000","last_clean":"2023-09-30T00:17:51.968754+0000","last_became_active":"2023-09-30T00:17:51.965549+0000","last_became_peered":"2023-09-30T00:17:51.965549+0000","last_unstale":"2023-09-30T00:17:51.968754+0000","last_undegraded":"2023-09-30T00:17:51.968754+0000","last_fullsized":"2023-09-30T00:17:51.968754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:30.526216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.953542+0000","last_change":"2023-09-30T00:17:51.596289+0000","last_active":"2023-09-30T00:17:51.953542+0000","last_peered":"2023-09-30T00:17:51.953542+0000","last_clean":"2023-09-30T00:17:51.953542+0000","last_became_active":"2023-09-30T00:17:51.595689+0000","last_became_peered":"2023-09-30T00:17:51.595689+0000","last_unstale":"2023-09-30T00:17:51.953542+0000","last_undegraded":"2023-09-30T00:17:51.953542+0000","last_fullsized":"2023-09-30T00:17:51.953542+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:46:55.016362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975889+0000","last_change":"2023-09-30T00:17:51.970420+0000","last_active":"2023-09-30T00:17:51.975889+0000","last_peered":"2023-09-30T00:17:51.975889+0000","last_clean":"2023-09-30T00:17:51.975889+0000","last_became_active":"2023-09-30T00:17:51.963999+0000","last_became_peered":"2023-09-30T00:17:51.963999+0000","last_unstale":"2023-09-30T00:17:51.975889+0000","last_undegraded":"2023-09-30T00:17:51.975889+0000","last_fullsized":"2023-09-30T00:17:51.975889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962989+0000","last_change":"2023-09-30T00:17:51.590485+0000","last_active":"2023-09-30T00:17:51.962989+0000","last_peered":"2023-09-30T00:17:51.962989+0000","last_clean":"2023-09-30T00:17:51.962989+0000","last_became_active":"2023-09-30T00:17:51.590210+0000","last_became_peered":"2023-09-30T00:17:51.590210+0000","last_unstale":"2023-09-30T00:17:51.962989+0000","last_undegraded":"2023-09-30T00:17:51.962989+0000","last_fullsized":"2023-09-30T00:17:51.962989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:32.708671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.944019+0000","last_change":"2023-09-30T00:17:45.196688+0000","last_active":"2023-09-30T00:17:51.944019+0000","last_peered":"2023-09-30T00:17:51.944019+0000","last_clean":"2023-09-30T00:17:51.944019+0000","last_became_active":"2023-09-30T00:17:45.196361+0000","last_became_peered":"2023-09-30T00:17:45.196361+0000","last_unstale":"2023-09-30T00:17:51.944019+0000","last_undegraded":"2023-09-30T00:17:51.944019+0000","last_fullsized":"2023-09-30T00:17:51.944019+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977735+0000","last_change":"2023-09-30T00:17:51.974386+0000","last_active":"2023-09-30T00:17:51.977735+0000","last_peered":"2023-09-30T00:17:51.977735+0000","last_clean":"2023-09-30T00:17:51.977735+0000","last_became_active":"2023-09-30T00:17:51.973776+0000","last_became_peered":"2023-09-30T00:17:51.973776+0000","last_unstale":"2023-09-30T00:17:51.977735+0000","last_undegraded":"2023-09-30T00:17:51.977735+0000","last_fullsized":"2023-09-30T00:17:51.977735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:25:21.282436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943386+0000","last_change":"2023-09-30T00:17:45.203067+0000","last_active":"2023-09-30T00:17:51.943386+0000","last_peered":"2023-09-30T00:17:51.943386+0000","last_clean":"2023-09-30T00:17:51.943386+0000","last_became_active":"2023-09-30T00:17:45.202823+0000","last_became_peered":"2023-09-30T00:17:45.202823+0000","last_unstale":"2023-09-30T00:17:51.943386+0000","last_undegraded":"2023-09-30T00:17:51.943386+0000","last_fullsized":"2023-09-30T00:17:51.943386+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:26:01.195302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-09-30T00:17:52.947337+0000","last_change":"2023-09-30T00:17:51.592843+0000","last_active":"2023-09-30T00:17:52.947337+0000","last_peered":"2023-09-30T00:17:52.947337+0000","last_clean":"2023-09-30T00:17:52.947337+0000","last_became_active":"2023-09-30T00:17:51.592601+0000","last_became_peered":"2023-09-30T00:17:51.592601+0000","last_unstale":"2023-09-30T00:17:52.947337+0000","last_undegraded":"2023-09-30T00:17:52.947337+0000","last_fullsized":"2023-09-30T00:17:52.947337+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968789+0000","last_change":"2023-09-30T00:17:51.965286+0000","last_active":"2023-09-30T00:17:51.968789+0000","last_peered":"2023-09-30T00:17:51.968789+0000","last_clean":"2023-09-30T00:17:51.968789+0000","last_became_active":"2023-09-30T00:17:51.964869+0000","last_became_peered":"2023-09-30T00:17:51.964869+0000","last_unstale":"2023-09-30T00:17:51.968789+0000","last_undegraded":"2023-09-30T00:17:51.968789+0000","last_fullsized":"2023-09-30T00:17:51.968789+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:46.192135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943060+0000","last_change":"2023-09-30T00:17:45.195737+0000","last_active":"2023-09-30T00:17:51.943060+0000","last_peered":"2023-09-30T00:17:51.943060+0000","last_clean":"2023-09-30T00:17:51.943060+0000","last_became_active":"2023-09-30T00:17:45.195464+0000","last_became_peered":"2023-09-30T00:17:45.195464+0000","last_unstale":"2023-09-30T00:17:51.943060+0000","last_undegraded":"2023-09-30T00:17:51.943060+0000","last_fullsized":"2023-09-30T00:17:51.943060+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:24:48.730978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952759+0000","last_change":"2023-09-30T00:17:47.199132+0000","last_active":"2023-09-30T00:17:51.952759+0000","last_peered":"2023-09-30T00:17:51.952759+0000","last_clean":"2023-09-30T00:17:51.952759+0000","last_became_active":"2023-09-30T00:17:47.198255+0000","last_became_peered":"2023-09-30T00:17:47.198255+0000","last_unstale":"2023-09-30T00:17:51.952759+0000","last_undegraded":"2023-09-30T00:17:51.952759+0000","last_fullsized":"2023-09-30T00:17:51.952759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_clean_scrub_stamp":"2023-09-30T00:17:46.189460+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:09.834201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943061+0000","last_change":"2023-09-30T00:17:51.575046+0000","last_active":"2023-09-30T00:17:51.943061+0000","last_peered":"2023-09-30T00:17:51.943061+0000","last_clean":"2023-09-30T00:17:51.943061+0000","last_became_active":"2023-09-30T00:17:51.574815+0000","last_became_peered":"2023-09-30T00:17:51.574815+0000","last_unstale":"2023-09-30T00:17:51.943061+0000","last_undegraded":"2023-09-30T00:17:51.943061+0000","last_fullsized":"2023-09-30T00:17:51.943061+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:43:57.892326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973520+0000","last_change":"2023-09-30T00:17:51.969276+0000","last_active":"2023-09-30T00:17:51.973520+0000","last_peered":"2023-09-30T00:17:51.973520+0000","last_clean":"2023-09-30T00:17:51.973520+0000","last_became_active":"2023-09-30T00:17:51.966513+0000","last_became_peered":"2023-09-30T00:17:51.966513+0000","last_unstale":"2023-09-30T00:17:51.973520+0000","last_undegraded":"2023-09-30T00:17:51.973520+0000","last_fullsized":"2023-09-30T00:17:51.973520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.946321+0000","last_change":"2023-09-30T00:17:45.197230+0000","last_active":"2023-09-30T00:17:51.946321+0000","last_peered":"2023-09-30T00:17:51.946321+0000","last_clean":"2023-09-30T00:17:51.946321+0000","last_became_active":"2023-09-30T00:17:45.196528+0000","last_became_peered":"2023-09-30T00:17:45.196528+0000","last_unstale":"2023-09-30T00:17:51.946321+0000","last_undegraded":"2023-09-30T00:17:51.946321+0000","last_fullsized":"2023-09-30T00:17:51.946321+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941919+0000","last_change":"2023-09-30T00:17:51.575520+0000","last_active":"2023-09-30T00:17:51.941919+0000","last_peered":"2023-09-30T00:17:51.941919+0000","last_clean":"2023-09-30T00:17:51.941919+0000","last_became_active":"2023-09-30T00:17:51.575306+0000","last_became_peered":"2023-09-30T00:17:51.575306+0000","last_unstale":"2023-09-30T00:17:51.941919+0000","last_undegraded":"2023-09-30T00:17:51.941919+0000","last_fullsized":"2023-09-30T00:17:51.941919+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:31:32.127076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.959411+0000","last_change":"2023-09-30T00:17:51.958852+0000","last_active":"2023-09-30T00:17:51.959411+0000","last_peered":"2023-09-30T00:17:51.959411+0000","last_clean":"2023-09-30T00:17:51.959411+0000","last_became_active":"2023-09-30T00:17:51.958570+0000","last_became_peered":"2023-09-30T00:17:51.958570+0000","last_unstale":"2023-09-30T00:17:51.959411+0000","last_undegraded":"2023-09-30T00:17:51.959411+0000","last_fullsized":"2023-09-30T00:17:51.959411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:34:27.611487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.940896+0000","last_change":"2023-09-30T00:17:45.196542+0000","last_active":"2023-09-30T00:17:51.940896+0000","last_peered":"2023-09-30T00:17:51.940896+0000","last_clean":"2023-09-30T00:17:51.940896+0000","last_became_active":"2023-09-30T00:17:45.196123+0000","last_became_peered":"2023-09-30T00:17:45.196123+0000","last_unstale":"2023-09-30T00:17:51.940896+0000","last_undegraded":"2023-09-30T00:17:51.940896+0000","last_fullsized":"2023-09-30T00:17:51.940896+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952371+0000","last_change":"2023-09-30T00:17:51.591518+0000","last_active":"2023-09-30T00:17:51.952371+0000","last_peered":"2023-09-30T00:17:51.952371+0000","last_clean":"2023-09-30T00:17:51.952371+0000","last_became_active":"2023-09-30T00:17:51.591253+0000","last_became_peered":"2023-09-30T00:17:51.591253+0000","last_unstale":"2023-09-30T00:17:51.952371+0000","last_undegraded":"2023-09-30T00:17:51.952371+0000","last_fullsized":"2023-09-30T00:17:51.952371+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:52:46.330854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980323+0000","last_change":"2023-09-30T00:17:51.976800+0000","last_active":"2023-09-30T00:17:51.980323+0000","last_peered":"2023-09-30T00:17:51.980323+0000","last_clean":"2023-09-30T00:17:51.980323+0000","last_became_active":"2023-09-30T00:17:51.976602+0000","last_became_peered":"2023-09-30T00:17:51.976602+0000","last_unstale":"2023-09-30T00:17:51.980323+0000","last_undegraded":"2023-09-30T00:17:51.980323+0000","last_fullsized":"2023-09-30T00:17:51.980323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:53:26.550299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952216+0000","last_change":"2023-09-30T00:17:45.207123+0000","last_active":"2023-09-30T00:17:51.952216+0000","last_peered":"2023-09-30T00:17:51.952216+0000","last_clean":"2023-09-30T00:17:51.952216+0000","last_became_active":"2023-09-30T00:17:45.206750+0000","last_became_peered":"2023-09-30T00:17:45.206750+0000","last_unstale":"2023-09-30T00:17:51.952216+0000","last_undegraded":"2023-09-30T00:17:51.952216+0000","last_fullsized":"2023-09-30T00:17:51.952216+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.945122+0000","last_change":"2023-09-30T00:17:51.585515+0000","last_active":"2023-09-30T00:17:51.945122+0000","last_peered":"2023-09-30T00:17:51.945122+0000","last_clean":"2023-09-30T00:17:51.945122+0000","last_became_active":"2023-09-30T00:17:51.585262+0000","last_became_peered":"2023-09-30T00:17:51.585262+0000","last_unstale":"2023-09-30T00:17:51.945122+0000","last_undegraded":"2023-09-30T00:17:51.945122+0000","last_fullsized":"2023-09-30T00:17:51.945122+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:04.914282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960357+0000","last_change":"2023-09-30T00:17:51.959726+0000","last_active":"2023-09-30T00:17:51.960357+0000","last_peered":"2023-09-30T00:17:51.960357+0000","last_clean":"2023-09-30T00:17:51.960357+0000","last_became_active":"2023-09-30T00:17:51.959328+0000","last_became_peered":"2023-09-30T00:17:51.959328+0000","last_unstale":"2023-09-30T00:17:51.960357+0000","last_undegraded":"2023-09-30T00:17:51.960357+0000","last_fullsized":"2023-09-30T00:17:51.960357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:37:08.356297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.954829+0000","last_change":"2023-09-30T00:17:45.206971+0000","last_active":"2023-09-30T00:17:51.954829+0000","last_peered":"2023-09-30T00:17:51.954829+0000","last_clean":"2023-09-30T00:17:51.954829+0000","last_became_active":"2023-09-30T00:17:45.206693+0000","last_became_peered":"2023-09-30T00:17:45.206693+0000","last_unstale":"2023-09-30T00:17:51.954829+0000","last_undegraded":"2023-09-30T00:17:51.954829+0000","last_fullsized":"2023-09-30T00:17:51.954829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943205+0000","last_change":"2023-09-30T00:17:51.575506+0000","last_active":"2023-09-30T00:17:51.943205+0000","last_peered":"2023-09-30T00:17:51.943205+0000","last_clean":"2023-09-30T00:17:51.943205+0000","last_became_active":"2023-09-30T00:17:51.574806+0000","last_became_peered":"2023-09-30T00:17:51.574806+0000","last_unstale":"2023-09-30T00:17:51.943205+0000","last_undegraded":"2023-09-30T00:17:51.943205+0000","last_fullsized":"2023-09-30T00:17:51.943205+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:51.816991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975408+0000","last_change":"2023-09-30T00:17:51.967785+0000","last_active":"2023-09-30T00:17:51.975408+0000","last_peered":"2023-09-30T00:17:51.975408+0000","last_clean":"2023-09-30T00:17:51.975408+0000","last_became_active":"2023-09-30T00:17:51.967419+0000","last_became_peered":"2023-09-30T00:17:51.967419+0000","last_unstale":"2023-09-30T00:17:51.975408+0000","last_undegraded":"2023-09-30T00:17:51.975408+0000","last_fullsized":"2023-09-30T00:17:51.975408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941618+0000","last_change":"2023-09-30T00:17:45.203202+0000","last_active":"2023-09-30T00:17:51.941618+0000","last_peered":"2023-09-30T00:17:51.941618+0000","last_clean":"2023-09-30T00:17:51.941618+0000","last_became_active":"2023-09-30T00:17:45.202879+0000","last_became_peered":"2023-09-30T00:17:45.202879+0000","last_unstale":"2023-09-30T00:17:51.941618+0000","last_undegraded":"2023-09-30T00:17:51.941618+0000","last_fullsized":"2023-09-30T00:17:51.941618+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:50:42.379943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.944043+0000","last_change":"2023-09-30T00:17:51.580771+0000","last_active":"2023-09-30T00:17:51.944043+0000","last_peered":"2023-09-30T00:17:51.944043+0000","last_clean":"2023-09-30T00:17:51.944043+0000","last_became_active":"2023-09-30T00:17:51.580495+0000","last_became_peered":"2023-09-30T00:17:51.580495+0000","last_unstale":"2023-09-30T00:17:51.944043+0000","last_undegraded":"2023-09-30T00:17:51.944043+0000","last_fullsized":"2023-09-30T00:17:51.944043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971832+0000","last_change":"2023-09-30T00:17:51.957745+0000","last_active":"2023-09-30T00:17:51.971832+0000","last_peered":"2023-09-30T00:17:51.971832+0000","last_clean":"2023-09-30T00:17:51.971832+0000","last_became_active":"2023-09-30T00:17:51.955331+0000","last_became_peered":"2023-09-30T00:17:51.955331+0000","last_unstale":"2023-09-30T00:17:51.971832+0000","last_undegraded":"2023-09-30T00:17:51.971832+0000","last_fullsized":"2023-09-30T00:17:51.971832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:45.393660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973877+0000","last_change":"2023-09-30T00:17:51.596021+0000","last_active":"2023-09-30T00:17:51.973877+0000","last_peered":"2023-09-30T00:17:51.973877+0000","last_clean":"2023-09-30T00:17:51.973877+0000","last_became_active":"2023-09-30T00:17:51.595745+0000","last_became_peered":"2023-09-30T00:17:51.595745+0000","last_unstale":"2023-09-30T00:17:51.973877+0000","last_undegraded":"2023-09-30T00:17:51.973877+0000","last_fullsized":"2023-09-30T00:17:51.973877+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:44.390191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960330+0000","last_change":"2023-09-30T00:17:51.959564+0000","last_active":"2023-09-30T00:17:51.960330+0000","last_peered":"2023-09-30T00:17:51.960330+0000","last_clean":"2023-09-30T00:17:51.960330+0000","last_became_active":"2023-09-30T00:17:51.959278+0000","last_became_peered":"2023-09-30T00:17:51.959278+0000","last_unstale":"2023-09-30T00:17:51.960330+0000","last_undegraded":"2023-09-30T00:17:51.960330+0000","last_fullsized":"2023-09-30T00:17:51.960330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:55.932612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974567+0000","last_change":"2023-09-30T00:17:51.587276+0000","last_active":"2023-09-30T00:17:51.974567+0000","last_peered":"2023-09-30T00:17:51.974567+0000","last_clean":"2023-09-30T00:17:51.974567+0000","last_became_active":"2023-09-30T00:17:51.586984+0000","last_became_peered":"2023-09-30T00:17:51.586984+0000","last_unstale":"2023-09-30T00:17:51.974567+0000","last_undegraded":"2023-09-30T00:17:51.974567+0000","last_fullsized":"2023-09-30T00:17:51.974567+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960802+0000","last_change":"2023-09-30T00:17:51.953787+0000","last_active":"2023-09-30T00:17:51.960802+0000","last_peered":"2023-09-30T00:17:51.960802+0000","last_clean":"2023-09-30T00:17:51.960802+0000","last_became_active":"2023-09-30T00:17:51.951805+0000","last_became_peered":"2023-09-30T00:17:51.951805+0000","last_unstale":"2023-09-30T00:17:51.960802+0000","last_undegraded":"2023-09-30T00:17:51.960802+0000","last_fullsized":"2023-09-30T00:17:51.960802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:57:04.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965610+0000","last_change":"2023-09-30T00:17:51.593225+0000","last_active":"2023-09-30T00:17:51.965610+0000","last_peered":"2023-09-30T00:17:51.965610+0000","last_clean":"2023-09-30T00:17:51.965610+0000","last_became_active":"2023-09-30T00:17:51.592913+0000","last_became_peered":"2023-09-30T00:17:51.592913+0000","last_unstale":"2023-09-30T00:17:51.965610+0000","last_undegraded":"2023-09-30T00:17:51.965610+0000","last_fullsized":"2023-09-30T00:17:51.965610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:37:56.776343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.961497+0000","last_change":"2023-09-30T00:17:51.955058+0000","last_active":"2023-09-30T00:17:51.961497+0000","last_peered":"2023-09-30T00:17:51.961497+0000","last_clean":"2023-09-30T00:17:51.961497+0000","last_became_active":"2023-09-30T00:17:51.954787+0000","last_became_peered":"2023-09-30T00:17:51.954787+0000","last_unstale":"2023-09-30T00:17:51.961497+0000","last_undegraded":"2023-09-30T00:17:51.961497+0000","last_fullsized":"2023-09-30T00:17:51.961497+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:49:30.148218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960867+0000","last_change":"2023-09-30T00:17:51.586229+0000","last_active":"2023-09-30T00:17:51.960867+0000","last_peered":"2023-09-30T00:17:51.960867+0000","last_clean":"2023-09-30T00:17:51.960867+0000","last_became_active":"2023-09-30T00:17:51.586027+0000","last_became_peered":"2023-09-30T00:17:51.586027+0000","last_unstale":"2023-09-30T00:17:51.960867+0000","last_undegraded":"2023-09-30T00:17:51.960867+0000","last_fullsized":"2023-09-30T00:17:51.960867+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:48:13.817514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962963+0000","last_change":"2023-09-30T00:17:51.959628+0000","last_active":"2023-09-30T00:17:51.962963+0000","last_peered":"2023-09-30T00:17:51.962963+0000","last_clean":"2023-09-30T00:17:51.962963+0000","last_became_active":"2023-09-30T00:17:51.958974+0000","last_became_peered":"2023-09-30T00:17:51.958974+0000","last_unstale":"2023-09-30T00:17:51.962963+0000","last_undegraded":"2023-09-30T00:17:51.962963+0000","last_fullsized":"2023-09-30T00:17:51.962963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:28.604134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"18'4","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960480+0000","last_change":"2023-09-30T00:17:51.590387+0000","last_active":"2023-09-30T00:17:51.960480+0000","last_peered":"2023-09-30T00:17:51.960480+0000","last_clean":"2023-09-30T00:17:51.960480+0000","last_became_active":"2023-09-30T00:17:51.585479+0000","last_became_peered":"2023-09-30T00:17:51.585479+0000","last_unstale":"2023-09-30T00:17:51.960480+0000","last_undegraded":"2023-09-30T00:17:51.960480+0000","last_fullsized":"2023-09-30T00:17:51.960480+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.959772+0000","last_change":"2023-09-30T00:17:51.949418+0000","last_active":"2023-09-30T00:17:51.959772+0000","last_peered":"2023-09-30T00:17:51.959772+0000","last_clean":"2023-09-30T00:17:51.959772+0000","last_became_active":"2023-09-30T00:17:51.949101+0000","last_became_peered":"2023-09-30T00:17:51.949101+0000","last_unstale":"2023-09-30T00:17:51.959772+0000","last_undegraded":"2023-09-30T00:17:51.959772+0000","last_fullsized":"2023-09-30T00:17:51.959772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975274+0000","last_change":"2023-09-30T00:17:51.964155+0000","last_active":"2023-09-30T00:17:51.975274+0000","last_peered":"2023-09-30T00:17:51.975274+0000","last_clean":"2023-09-30T00:17:51.975274+0000","last_became_active":"2023-09-30T00:17:51.958334+0000","last_became_peered":"2023-09-30T00:17:51.958334+0000","last_unstale":"2023-09-30T00:17:51.975274+0000","last_undegraded":"2023-09-30T00:17:51.975274+0000","last_fullsized":"2023-09-30T00:17:51.975274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.959645+0000","last_change":"2023-09-30T00:17:51.959066+0000","last_active":"2023-09-30T00:17:51.959645+0000","last_peered":"2023-09-30T00:17:51.959645+0000","last_clean":"2023-09-30T00:17:51.959645+0000","last_became_active":"2023-09-30T00:17:51.958741+0000","last_became_peered":"2023-09-30T00:17:51.958741+0000","last_unstale":"2023-09-30T00:17:51.959645+0000","last_undegraded":"2023-09-30T00:17:51.959645+0000","last_fullsized":"2023-09-30T00:17:51.959645+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:54:35.602823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976104+0000","last_change":"2023-09-30T00:17:51.970791+0000","last_active":"2023-09-30T00:17:51.976104+0000","last_peered":"2023-09-30T00:17:51.976104+0000","last_clean":"2023-09-30T00:17:51.976104+0000","last_became_active":"2023-09-30T00:17:51.964142+0000","last_became_peered":"2023-09-30T00:17:51.964142+0000","last_unstale":"2023-09-30T00:17:51.976104+0000","last_undegraded":"2023-09-30T00:17:51.976104+0000","last_fullsized":"2023-09-30T00:17:51.976104+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.967656+0000","last_change":"2023-09-30T00:17:51.967167+0000","last_active":"2023-09-30T00:17:51.967656+0000","last_peered":"2023-09-30T00:17:51.967656+0000","last_clean":"2023-09-30T00:17:51.967656+0000","last_became_active":"2023-09-30T00:17:51.966937+0000","last_became_peered":"2023-09-30T00:17:51.966937+0000","last_unstale":"2023-09-30T00:17:51.967656+0000","last_undegraded":"2023-09-30T00:17:51.967656+0000","last_fullsized":"2023-09-30T00:17:51.967656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:32:11.138421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960344+0000","last_change":"2023-09-30T00:17:51.959728+0000","last_active":"2023-09-30T00:17:51.960344+0000","last_peered":"2023-09-30T00:17:51.960344+0000","last_clean":"2023-09-30T00:17:51.960344+0000","last_became_active":"2023-09-30T00:17:51.959512+0000","last_became_peered":"2023-09-30T00:17:51.959512+0000","last_unstale":"2023-09-30T00:17:51.960344+0000","last_undegraded":"2023-09-30T00:17:51.960344+0000","last_fullsized":"2023-09-30T00:17:51.960344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:05:15.182292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9232,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362608,"statfs":{"total":96636764160,"available":96627310592,"internally_reserved":0,"allocated":794624,"data_stored":651535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8072,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363768,"statfs":{"total":96636764160,"available":96628498432,"internally_reserved":0,"allocated":196608,"data_stored":60391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9084,"kb_used_data":756,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94362756,"statfs":{"total":96636764160,"available":96627462144,"internally_reserved":0,"allocated":774144,"data_stored":650359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]}]},{"osd":1,"up_from":10,"seq":42949672966,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8064,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363776,"statfs":{"total":96636764160,"available":96628506624,"internally_reserved":0,"allocated":188416,"data_stored":60083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:18:00.816 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-09-30T00:18:00.816 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T00:18:00.816 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T00:18:00.817 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T00:18:00.817 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T00:18:00.964 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77b1aab700 1 Processor -- start 2023-09-30T00:18:00.964 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77b1aab700 1 -- start start 2023-09-30T00:18:00.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77b1aab700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 0x7f77ac0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:00.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77b1aab700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f77ac0b7160 0x7f77ac0a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:00.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77b1aab700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f77ac13fd70 0x7f77ac1401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:00.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77b1aab700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ac0a1310 con 0x7f77ac0b7160 2023-09-30T00:18:00.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77b1aab700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ac071020 con 0x7f77ac13fd70 2023-09-30T00:18:00.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77b1aab700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac0b76a0 con 0x7f77ac070c50 2023-09-30T00:18:00.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77aaffd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f77ac0b7160 0x7f77ac0a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:00.967 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77ab7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 0x7f77ac0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:00.968 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77ab7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 0x7f77ac0a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39108/0 (socket says 172.21.15.94:39108) 2023-09-30T00:18:00.968 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77aaffd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f77ac0b7160 0x7f77ac0a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50606/0 (socket says 172.21.15.94:50606) 2023-09-30T00:18:00.968 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.964+0000 7f77ab7fe700 1 -- 172.21.15.94:0/3626148837 learned_addr learned my addr 172.21.15.94:0/3626148837 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:00.968 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77abfff700 1 --2- 172.21.15.94:0/3626148837 >> v2:172.21.15.121:3300/0 conn(0x7f77ac13fd70 0x7f77ac1401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:00.968 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77ab7fe700 1 -- 172.21.15.94:0/3626148837 >> v2:172.21.15.121:3300/0 conn(0x7f77ac13fd70 msgr2=0x7f77ac1401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:00.968 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77ab7fe700 1 --2- 172.21.15.94:0/3626148837 >> v2:172.21.15.121:3300/0 conn(0x7f77ac13fd70 0x7f77ac1401d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.968 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77ab7fe700 1 -- 172.21.15.94:0/3626148837 >> v2:172.21.15.94:3300/0 conn(0x7f77ac0b7160 msgr2=0x7f77ac0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:00.968 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77ab7fe700 1 --2- 172.21.15.94:0/3626148837 >> v2:172.21.15.94:3300/0 conn(0x7f77ac0b7160 0x7f77ac0a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.968 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77ab7fe700 1 -- 172.21.15.94:0/3626148837 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac0b7940 con 0x7f77ac070c50 2023-09-30T00:18:00.968 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77aaffd700 1 --2- 172.21.15.94:0/3626148837 >> v2:172.21.15.94:3300/0 conn(0x7f77ac0b7160 0x7f77ac0a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:00.968 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77ab7fe700 1 --2- 172.21.15.94:0/3626148837 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 0x7f77ac0a0dd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f77ac067c90 tx=0x7f779c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef0e9fb995f9df78 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:00.969 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77aa7fc700 1 -- 172.21.15.94:0/3626148837 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c00b6d0 con 0x7f77ac070c50 2023-09-30T00:18:00.969 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77aa7fc700 1 -- 172.21.15.94:0/3626148837 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779c00b830 con 0x7f77ac070c50 2023-09-30T00:18:00.969 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77aa7fc700 1 -- 172.21.15.94:0/3626148837 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c00bbf0 con 0x7f77ac070c50 2023-09-30T00:18:00.969 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77b1aab700 1 -- 172.21.15.94:0/3626148837 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 msgr2=0x7f77ac0a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:00.969 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77b1aab700 1 --2- 172.21.15.94:0/3626148837 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 0x7f77ac0a0dd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f77ac067c90 tx=0x7f779c009250 comp rx=0 tx=0).stop 2023-09-30T00:18:00.969 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.965+0000 7f77b1aab700 1 -- 172.21.15.94:0/3626148837 shutdown_connections 2023-09-30T00:18:00.969 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.966+0000 7f77b1aab700 1 --2- 172.21.15.94:0/3626148837 >> v2:172.21.15.121:3300/0 conn(0x7f77ac13fd70 0x7f77ac1401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.969 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.966+0000 7f77b1aab700 1 --2- 172.21.15.94:0/3626148837 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 0x7f77ac0a0dd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.969 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.966+0000 7f77b1aab700 1 --2- 172.21.15.94:0/3626148837 >> v2:172.21.15.94:3300/0 conn(0x7f77ac0b7160 0x7f77ac0a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.969 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.966+0000 7f77b1aab700 1 -- 172.21.15.94:0/3626148837 >> 172.21.15.94:0/3626148837 conn(0x7f77ac06a090 msgr2=0x7f77ac0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:00.969 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.966+0000 7f77b1aab700 1 -- 172.21.15.94:0/3626148837 shutdown_connections 2023-09-30T00:18:00.969 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.966+0000 7f77b1aab700 1 -- 172.21.15.94:0/3626148837 wait complete. 2023-09-30T00:18:00.970 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77b1aab700 1 Processor -- start 2023-09-30T00:18:00.970 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77b1aab700 1 -- start start 2023-09-30T00:18:00.972 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77b1aab700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 0x7f77ac143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:00.972 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77b1aab700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ac0b7160 0x7f77ac150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:00.972 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77b1aab700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77ac13fd70 0x7f77ac0ab830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:00.972 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77b1aab700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac0abd70 con 0x7f77ac13fd70 2023-09-30T00:18:00.973 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77b1aab700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac150c60 con 0x7f77ac0b7160 2023-09-30T00:18:00.973 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77b1aab700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac150f20 con 0x7f77ac070c50 2023-09-30T00:18:00.973 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77ab7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 0x7f77ac143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:00.973 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77abfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77ac13fd70 0x7f77ac0ab830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:00.973 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77ab7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 0x7f77ac143fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39120/0 (socket says 172.21.15.94:39120) 2023-09-30T00:18:00.973 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77ab7fe700 1 -- 172.21.15.94:0/2882635483 learned_addr learned my addr 172.21.15.94:0/2882635483 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:00.973 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77aaffd700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ac0b7160 0x7f77ac150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:00.973 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77ab7fe700 1 -- 172.21.15.94:0/2882635483 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ac0b7160 msgr2=0x7f77ac150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:00.973 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77ab7fe700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ac0b7160 0x7f77ac150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.973 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77ab7fe700 1 -- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77ac13fd70 msgr2=0x7f77ac0ab830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:00.973 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.967+0000 7f77ab7fe700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77ac13fd70 0x7f77ac0ab830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:00.974 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.968+0000 7f77ab7fe700 1 -- 172.21.15.94:0/2882635483 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a0009420 con 0x7f77ac070c50 2023-09-30T00:18:00.974 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.968+0000 7f77abfff700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77ac13fd70 0x7f77ac0ab830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:00.974 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.968+0000 7f77ab7fe700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 0x7f77ac143fc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f779c006fd0 tx=0x7f779c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:00.974 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.968+0000 7f77937fe700 1 -- 172.21.15.94:0/2882635483 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c000d10 con 0x7f77ac070c50 2023-09-30T00:18:00.974 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.968+0000 7f77937fe700 1 -- 172.21.15.94:0/2882635483 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779c00bbb0 con 0x7f77ac070c50 2023-09-30T00:18:00.974 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.968+0000 7f77b1aab700 1 -- 172.21.15.94:0/2882635483 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779c0094f0 con 0x7f77ac070c50 2023-09-30T00:18:00.974 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.968+0000 7f77937fe700 1 -- 172.21.15.94:0/2882635483 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c011670 con 0x7f77ac070c50 2023-09-30T00:18:00.974 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.968+0000 7f77b1aab700 1 -- 172.21.15.94:0/2882635483 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ac13a300 con 0x7f77ac070c50 2023-09-30T00:18:00.974 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.969+0000 7f77b1aab700 1 -- 172.21.15.94:0/2882635483 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ac071630 con 0x7f77ac070c50 2023-09-30T00:18:00.974 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.970+0000 7f77937fe700 1 -- 172.21.15.94:0/2882635483 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f779c0119e0 con 0x7f77ac070c50 2023-09-30T00:18:00.974 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.970+0000 7f77937fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:00.975 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.970+0000 7f77937fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:00.975 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.970+0000 7f77937fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:00.975 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.970+0000 7f77937fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:00.975 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.970+0000 7f77937fe700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7798098780 0x7f779809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:00.975 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.970+0000 7f77aaffd700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7798098780 0x7f779809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:00.975 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.970+0000 7f77937fe700 1 -- 172.21.15.94:0/2882635483 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f779c0c20e0 con 0x7f77ac070c50 2023-09-30T00:18:00.975 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.971+0000 7f77aaffd700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7798098780 0x7f779809ac30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f77ac0aad70 tx=0x7f77a00091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:00.975 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:00.973+0000 7f77937fe700 1 -- 172.21.15.94:0/2882635483 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f779c08f820 con 0x7f77ac070c50 2023-09-30T00:18:01.205 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.205+0000 7f77b1aab700 1 -- 172.21.15.94:0/2882635483 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f77ac13f5e0 con 0x7f77ac070c50 2023-09-30T00:18:01.209 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.209+0000 7f77937fe700 1 -- 172.21.15.94:0/2882635483 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+244962 (secure 0 0 0) 0x7f779c092e40 con 0x7f77ac070c50 2023-09-30T00:18:01.212 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:01.214 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.212+0000 7f77b1aab700 10 mgrc shutdown 2023-09-30T00:18:01.214 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.212+0000 7f77b1aab700 1 -- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7798098780 msgr2=0x7f779809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.214 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.212+0000 7f77b1aab700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7798098780 0x7f779809ac30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f77ac0aad70 tx=0x7f77a00091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.215 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.212+0000 7f77b1aab700 1 -- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 msgr2=0x7f77ac143fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.215 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.212+0000 7f77b1aab700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 0x7f77ac143fc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f779c006fd0 tx=0x7f779c017000 comp rx=0 tx=0).stop 2023-09-30T00:18:01.215 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.212+0000 7f77b1aab700 1 -- 172.21.15.94:0/2882635483 shutdown_connections 2023-09-30T00:18:01.215 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.212+0000 7f77b1aab700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7798098780 0x7f779809ac30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.215 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.212+0000 7f77b1aab700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f77ac070c50 0x7f77ac143fc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.215 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.212+0000 7f77b1aab700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77ac13fd70 0x7f77ac0ab830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.215 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.212+0000 7f77b1aab700 1 --2- 172.21.15.94:0/2882635483 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ac0b7160 0x7f77ac150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.215 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.212+0000 7f77b1aab700 1 -- 172.21.15.94:0/2882635483 >> 172.21.15.94:0/2882635483 conn(0x7f77ac06a090 msgr2=0x7f77ac13ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:01.215 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.212+0000 7f77b1aab700 1 -- 172.21.15.94:0/2882635483 shutdown_connections 2023-09-30T00:18:01.215 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.213+0000 7f77b1aab700 1 -- 172.21.15.94:0/2882635483 wait complete. 2023-09-30T00:18:01.224 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":5,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":84242},{"type":"v1","addr":"172.21.15.94:6809","nonce":84242}]},"active_addr":"172.21.15.94:6809/84242","active_change":"2023-09-30T00:17:39.844110+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4100,"name":"z","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":4105,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":2439891298}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":3749920472}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":1691216602}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":2232646692}]}]} 2023-09-30T00:18:01.230 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T00:18:01.230 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T00:18:01.231 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:18:01.379 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.379+0000 7fe5ae129700 1 Processor -- start 2023-09-30T00:18:01.379 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.379+0000 7fe5ae129700 1 -- start start 2023-09-30T00:18:01.382 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.379+0000 7fe5ae129700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe5a8070c50 0x7fe5a80a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.382 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.379+0000 7fe5ae129700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe5a80b7160 0x7fe5a80a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.382 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.379+0000 7fe5ae129700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a80a1310 con 0x7fe5a813fd70 2023-09-30T00:18:01.382 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.379+0000 7fe5ae129700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a8071020 con 0x7fe5a8070c50 2023-09-30T00:18:01.382 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.379+0000 7fe5ae129700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a80b76a0 con 0x7fe5a80b7160 2023-09-30T00:18:01.382 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.379+0000 7fe5a77fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe5a8070c50 0x7fe5a80a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.382 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.379+0000 7fe5a77fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe5a8070c50 0x7fe5a80a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50612/0 (socket says 172.21.15.94:50612) 2023-09-30T00:18:01.382 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.379+0000 7fe5a77fe700 1 -- 172.21.15.94:0/3726944628 learned_addr learned my addr 172.21.15.94:0/3726944628 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:01.382 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.379+0000 7fe5a7fff700 1 --1- 172.21.15.94:0/3726944628 >> v1:172.21.15.94:6789/0 conn(0x7fe5a813fd70 0x7fe5a80b4dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50208/0 (socket says 172.21.15.94:50208) 2023-09-30T00:18:01.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5a6ffd700 1 --2- 172.21.15.94:0/3726944628 >> v2:172.21.15.121:3300/0 conn(0x7fe5a80b7160 0x7fe5a80a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5a77fe700 1 -- 172.21.15.94:0/3726944628 >> v2:172.21.15.121:3300/0 conn(0x7fe5a80b7160 msgr2=0x7fe5a80a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:01.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5a77fe700 1 --2- 172.21.15.94:0/3726944628 >> v2:172.21.15.121:3300/0 conn(0x7fe5a80b7160 0x7fe5a80a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5a77fe700 1 -- 172.21.15.94:0/3726944628 >> v1:172.21.15.94:6789/0 conn(0x7fe5a813fd70 legacy=0x7fe5a80b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5a77fe700 1 -- 172.21.15.94:0/3726944628 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a80b7940 con 0x7fe5a8070c50 2023-09-30T00:18:01.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5a67fc700 1 -- 172.21.15.94:0/3726944628 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467006996 0 0) 0x7fe598001110 con 0x7fe5a813fd70 2023-09-30T00:18:01.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5a6ffd700 1 --2- 172.21.15.94:0/3726944628 >> v2:172.21.15.121:3300/0 conn(0x7fe5a80b7160 0x7fe5a80a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:01.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5a77fe700 1 --2- 172.21.15.94:0/3726944628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a8070c50 0x7fe5a80a0dd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5a806a880 tx=0x7fe5a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=281c062bef42f929 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:01.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5a67fc700 1 -- 172.21.15.94:0/3726944628 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a000e070 con 0x7fe5a8070c50 2023-09-30T00:18:01.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5a67fc700 1 -- 172.21.15.94:0/3726944628 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a000bae0 con 0x7fe5a8070c50 2023-09-30T00:18:01.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5a67fc700 1 -- 172.21.15.94:0/3726944628 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a00092e0 con 0x7fe5a8070c50 2023-09-30T00:18:01.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5ae129700 1 -- 172.21.15.94:0/3726944628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a8070c50 msgr2=0x7fe5a80a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.380+0000 7fe5ae129700 1 --2- 172.21.15.94:0/3726944628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a8070c50 0x7fe5a80a0dd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5a806a880 tx=0x7fe5a0009840 comp rx=0 tx=0).stop 2023-09-30T00:18:01.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.381+0000 7fe5ae129700 1 -- 172.21.15.94:0/3726944628 shutdown_connections 2023-09-30T00:18:01.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.381+0000 7fe5ae129700 1 --2- 172.21.15.94:0/3726944628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a8070c50 0x7fe5a80a0dd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.381+0000 7fe5ae129700 1 --2- 172.21.15.94:0/3726944628 >> v2:172.21.15.121:3300/0 conn(0x7fe5a80b7160 0x7fe5a80a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.381+0000 7fe5ae129700 1 -- 172.21.15.94:0/3726944628 >> 172.21.15.94:0/3726944628 conn(0x7fe5a806a090 msgr2=0x7fe5a80702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:01.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.381+0000 7fe5ae129700 1 -- 172.21.15.94:0/3726944628 shutdown_connections 2023-09-30T00:18:01.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.381+0000 7fe5ae129700 1 -- 172.21.15.94:0/3726944628 wait complete. 2023-09-30T00:18:01.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5ae129700 1 Processor -- start 2023-09-30T00:18:01.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5ae129700 1 -- start start 2023-09-30T00:18:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5ae129700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5a8070c50 0x7fe5a8151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5ae129700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a80b7160 0x7fe5a81421f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5ae129700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5a813fd70 0x7fe5a8142730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5ae129700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a8150a70 con 0x7fe5a80b7160 2023-09-30T00:18:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5ae129700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a8150d30 con 0x7fe5a8070c50 2023-09-30T00:18:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5ae129700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80a54b0 con 0x7fe5a813fd70 2023-09-30T00:18:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5a6ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a80b7160 0x7fe5a81421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5a7fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5a813fd70 0x7fe5a8142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5a6ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a80b7160 0x7fe5a81421f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50628/0 (socket says 172.21.15.94:50628) 2023-09-30T00:18:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5a6ffd700 1 -- 172.21.15.94:0/233879067 learned_addr learned my addr 172.21.15.94:0/233879067 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.382+0000 7fe5a7fff700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5a813fd70 0x7fe5a8142730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39130/0 (socket says 172.21.15.94:39130) 2023-09-30T00:18:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.383+0000 7fe5a77fe700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5a8070c50 0x7fe5a8151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.383+0000 7fe5a6ffd700 1 -- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5a813fd70 msgr2=0x7fe5a8142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.383+0000 7fe5a6ffd700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5a813fd70 0x7fe5a8142730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.383+0000 7fe5a6ffd700 1 -- 172.21.15.94:0/233879067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5a8070c50 msgr2=0x7fe5a8151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.383+0000 7fe5a6ffd700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5a8070c50 0x7fe5a8151280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.383+0000 7fe5a6ffd700 1 -- 172.21.15.94:0/233879067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a0009ae0 con 0x7fe5a80b7160 2023-09-30T00:18:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.383+0000 7fe5a6ffd700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a80b7160 0x7fe5a81421f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe59000e8d0 tx=0x7fe590008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.383+0000 7fe59f7fe700 1 -- 172.21.15.94:0/233879067 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590013410 con 0x7fe5a80b7160 2023-09-30T00:18:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.383+0000 7fe59f7fe700 1 -- 172.21.15.94:0/233879067 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe590013570 con 0x7fe5a80b7160 2023-09-30T00:18:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.383+0000 7fe5ae129700 1 -- 172.21.15.94:0/233879067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a80a6180 con 0x7fe5a80b7160 2023-09-30T00:18:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.383+0000 7fe59f7fe700 1 -- 172.21.15.94:0/233879067 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590013910 con 0x7fe5a80b7160 2023-09-30T00:18:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.383+0000 7fe5ae129700 1 -- 172.21.15.94:0/233879067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a81507b0 con 0x7fe5a80b7160 2023-09-30T00:18:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.384+0000 7fe5ae129700 1 -- 172.21.15.94:0/233879067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a81431f0 con 0x7fe5a80b7160 2023-09-30T00:18:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.385+0000 7fe59f7fe700 1 -- 172.21.15.94:0/233879067 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe590013d00 con 0x7fe5a80b7160 2023-09-30T00:18:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.385+0000 7fe59f7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.385+0000 7fe59f7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.385+0000 7fe59f7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.385+0000 7fe59f7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.385+0000 7fe59f7fe700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe588098780 0x7fe58809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.385+0000 7fe5a77fe700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe588098780 0x7fe58809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.385+0000 7fe59f7fe700 1 -- 172.21.15.94:0/233879067 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fe5900c01f0 con 0x7fe5a80b7160 2023-09-30T00:18:01.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.386+0000 7fe5a77fe700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe588098780 0x7fe58809ac30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009880 tx=0x7fe5a0018000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:01.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.389+0000 7fe59f7fe700 1 -- 172.21.15.94:0/233879067 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe59008d8a0 con 0x7fe5a80b7160 2023-09-30T00:18:01.577 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.576+0000 7fe5ae129700 1 -- 172.21.15.94:0/233879067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5a8141ff0 con 0x7fe5a80b7160 2023-09-30T00:18:01.577 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.577+0000 7fe59f7fe700 1 -- 172.21.15.94:0/233879067 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12197 (secure 0 0 0) 0x7fe590020090 con 0x7fe5a80b7160 2023-09-30T00:18:01.578 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:01.578 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":19,"fsid":"3c891173-587e-4c6f-9c6a-04ec750eedad","created":"2023-09-30T00:17:34.347592+0000","modified":"2023-09-30T00:17:52.938964+0000","last_up_change":"2023-09-30T00:17:42.879328+0000","last_in_change":"2023-09-30T00:17:35.583427+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T00:17:43.481613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T00:17:45.892643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-09-30T00:17:50.220028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-09-30T00:17:50.911066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c508b402-f8ce-4461-823b-9e35fcd8d40f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":84364},{"type":"v1","addr":"172.21.15.94:6801","nonce":84364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":84364},{"type":"v1","addr":"172.21.15.94:6803","nonce":84364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":84364},{"type":"v1","addr":"172.21.15.94:6807","nonce":84364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":84364},{"type":"v1","addr":"172.21.15.94:6805","nonce":84364}]},"public_addr":"172.21.15.94:6801/84364","cluster_addr":"172.21.15.94:6803/84364","heartbeat_back_addr":"172.21.15.94:6807/84364","heartbeat_front_addr":"172.21.15.94:6805/84364","state":["exists","up"]},{"osd":1,"uuid":"e96ecdf0-bb38-4eb1-be99-54f813c12888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":86850},{"type":"v1","addr":"172.21.15.121:6817","nonce":86850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":86850},{"type":"v1","addr":"172.21.15.121:6819","nonce":86850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":86850},{"type":"v1","addr":"172.21.15.121:6823","nonce":86850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":86850},{"type":"v1","addr":"172.21.15.121:6821","nonce":86850}]},"public_addr":"172.21.15.121:6817/86850","cluster_addr":"172.21.15.121:6819/86850","heartbeat_back_addr":"172.21.15.121:6823/86850","heartbeat_front_addr":"172.21.15.121:6821/86850","state":["exists","up"]},{"osd":2,"uuid":"68700ae4-764c-4dff-9cf2-5f33132cd0d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":86820},{"type":"v1","addr":"172.21.15.121:6801","nonce":86820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":86820},{"type":"v1","addr":"172.21.15.121:6803","nonce":86820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":86820},{"type":"v1","addr":"172.21.15.121:6810","nonce":86820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":86820},{"type":"v1","addr":"172.21.15.121:6806","nonce":86820}]},"public_addr":"172.21.15.121:6801/86820","cluster_addr":"172.21.15.121:6803/86820","heartbeat_back_addr":"172.21.15.121:6810/86820","heartbeat_front_addr":"172.21.15.121:6806/86820","state":["exists","up"]},{"osd":3,"uuid":"54f17ec8-2561-42e8-bbd7-262130ffdac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":86821},{"type":"v1","addr":"172.21.15.121:6807","nonce":86821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":86821},{"type":"v1","addr":"172.21.15.121:6811","nonce":86821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":86821},{"type":"v1","addr":"172.21.15.121:6815","nonce":86821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":86821},{"type":"v1","addr":"172.21.15.121:6813","nonce":86821}]},"public_addr":"172.21.15.121:6807/86821","cluster_addr":"172.21.15.121:6811/86821","heartbeat_back_addr":"172.21.15.121:6815/86821","heartbeat_front_addr":"172.21.15.121:6813/86821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:39.951865+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.769149+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.738994+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.748197+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:18:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 10 mgrc shutdown 2023-09-30T00:18:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 1 -- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe588098780 msgr2=0x7fe58809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe588098780 0x7fe58809ac30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009880 tx=0x7fe5a0018000 comp rx=0 tx=0).stop 2023-09-30T00:18:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 1 -- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a80b7160 msgr2=0x7fe5a81421f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a80b7160 0x7fe5a81421f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe59000e8d0 tx=0x7fe590008040 comp rx=0 tx=0).stop 2023-09-30T00:18:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 1 -- 172.21.15.94:0/233879067 shutdown_connections 2023-09-30T00:18:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe588098780 0x7fe58809ac30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.580 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5a813fd70 0x7fe5a8142730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.581 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5a8070c50 0x7fe5a8151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.581 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 1 --2- 172.21.15.94:0/233879067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a80b7160 0x7fe5a81421f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.581 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 1 -- 172.21.15.94:0/233879067 >> 172.21.15.94:0/233879067 conn(0x7fe5a806a090 msgr2=0x7fe5a80b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:01.581 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 1 -- 172.21.15.94:0/233879067 shutdown_connections 2023-09-30T00:18:01.581 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.579+0000 7fe5ae129700 1 -- 172.21.15.94:0/233879067 wait complete. 2023-09-30T00:18:01.591 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T00:18:01.592 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:18:01.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.704+0000 7ff71d816700 1 Processor -- start 2023-09-30T00:18:01.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.704+0000 7ff71d816700 1 -- start start 2023-09-30T00:18:01.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff71d816700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff718070c50 0x7ff7180a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff71d816700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff7180b7160 0x7ff7180a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff71d816700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff71813fd70 0x7ff7181401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff71d816700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7180a1310 con 0x7ff7180b7160 2023-09-30T00:18:01.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff71d816700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff718071020 con 0x7ff718070c50 2023-09-30T00:18:01.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff71d816700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7180b76a0 con 0x7ff71813fd70 2023-09-30T00:18:01.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff70e7fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff7180b7160 0x7ff7180a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff70e7fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff7180b7160 0x7ff7180a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50634/0 (socket says 172.21.15.94:50634) 2023-09-30T00:18:01.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff70e7fc700 1 -- 172.21.15.94:0/579174462 learned_addr learned my addr 172.21.15.94:0/579174462 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:01.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff7177fe700 1 --2- 172.21.15.94:0/579174462 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff71813fd70 0x7ff7181401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff716ffd700 1 --2- 172.21.15.94:0/579174462 >> v2:172.21.15.121:3300/0 conn(0x7ff718070c50 0x7ff7180a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff70e7fc700 1 -- 172.21.15.94:0/579174462 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff71813fd70 msgr2=0x7ff7181401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:01.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff70e7fc700 1 --2- 172.21.15.94:0/579174462 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff71813fd70 0x7ff7181401b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff70e7fc700 1 -- 172.21.15.94:0/579174462 >> v2:172.21.15.121:3300/0 conn(0x7ff718070c50 msgr2=0x7ff7180a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:01.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff70e7fc700 1 --2- 172.21.15.94:0/579174462 >> v2:172.21.15.121:3300/0 conn(0x7ff718070c50 0x7ff7180a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff70e7fc700 1 -- 172.21.15.94:0/579174462 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7180b7940 con 0x7ff7180b7160 2023-09-30T00:18:01.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff7177fe700 1 --2- 172.21.15.94:0/579174462 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff71813fd70 0x7ff7181401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff70e7fc700 1 --2- 172.21.15.94:0/579174462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7180b7160 0x7ff7180a2530 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff7180a9760 tx=0x7ff700009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d2313dcbf4ee8f5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff7167fc700 1 -- 172.21.15.94:0/579174462 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff700009e70 con 0x7ff7180b7160 2023-09-30T00:18:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff7167fc700 1 -- 172.21.15.94:0/579174462 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70000b8f0 con 0x7ff7180b7160 2023-09-30T00:18:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.705+0000 7ff7167fc700 1 -- 172.21.15.94:0/579174462 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70000bc90 con 0x7ff7180b7160 2023-09-30T00:18:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.706+0000 7ff71d816700 1 -- 172.21.15.94:0/579174462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7180b7160 msgr2=0x7ff7180a2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.706+0000 7ff71d816700 1 --2- 172.21.15.94:0/579174462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7180b7160 0x7ff7180a2530 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff7180a9760 tx=0x7ff700009250 comp rx=0 tx=0).stop 2023-09-30T00:18:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.706+0000 7ff71d816700 1 -- 172.21.15.94:0/579174462 shutdown_connections 2023-09-30T00:18:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.706+0000 7ff71d816700 1 --2- 172.21.15.94:0/579174462 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff71813fd70 0x7ff7181401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.706+0000 7ff71d816700 1 --2- 172.21.15.94:0/579174462 >> v2:172.21.15.121:3300/0 conn(0x7ff718070c50 0x7ff7180a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.706+0000 7ff71d816700 1 --2- 172.21.15.94:0/579174462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7180b7160 0x7ff7180a2530 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.706+0000 7ff71d816700 1 -- 172.21.15.94:0/579174462 >> 172.21.15.94:0/579174462 conn(0x7ff71806a090 msgr2=0x7ff7180702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:01.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.706+0000 7ff71d816700 1 -- 172.21.15.94:0/579174462 shutdown_connections 2023-09-30T00:18:01.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.706+0000 7ff71d816700 1 -- 172.21.15.94:0/579174462 wait complete. 2023-09-30T00:18:01.715 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.707+0000 7ff71d816700 1 Processor -- start 2023-09-30T00:18:01.715 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.707+0000 7ff71d816700 1 -- start start 2023-09-30T00:18:01.716 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff71d816700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff718070c50 0x7ff7180a88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.716 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff71d816700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff7180b7160 0x7ff7180a6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.716 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff71d816700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff71813fd70 0x7ff71814ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.716 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff71d816700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7180a5c20 con 0x7ff71813fd70 2023-09-30T00:18:01.716 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff71d816700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff718150e80 con 0x7ff718070c50 2023-09-30T00:18:01.716 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff71d816700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff718151140 con 0x7ff7180b7160 2023-09-30T00:18:01.716 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff7177fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff71813fd70 0x7ff71814ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.716 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff70e7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff7180b7160 0x7ff7180a6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.716 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff7177fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff71813fd70 0x7ff71814ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50646/0 (socket says 172.21.15.94:50646) 2023-09-30T00:18:01.716 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff7177fe700 1 -- 172.21.15.94:0/2504389266 learned_addr learned my addr 172.21.15.94:0/2504389266 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:01.716 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff70e7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff7180b7160 0x7ff7180a6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39150/0 (socket says 172.21.15.94:39150) 2023-09-30T00:18:01.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff716ffd700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff718070c50 0x7ff7180a88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff7177fe700 1 -- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff7180b7160 msgr2=0x7ff7180a6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff7177fe700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff7180b7160 0x7ff7180a6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff7177fe700 1 -- 172.21.15.94:0/2504389266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff718070c50 msgr2=0x7ff7180a88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff7177fe700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff718070c50 0x7ff7180a88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff7177fe700 1 -- 172.21.15.94:0/2504389266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff710009a20 con 0x7ff71813fd70 2023-09-30T00:18:01.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff7177fe700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff71813fd70 0x7ff71814ff30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff710009f20 tx=0x7ff71000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:01.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff70f7fe700 1 -- 172.21.15.94:0/2504389266 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71000cb70 con 0x7ff71813fd70 2023-09-30T00:18:01.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff70f7fe700 1 -- 172.21.15.94:0/2504389266 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71000f040 con 0x7ff71813fd70 2023-09-30T00:18:01.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff71d816700 1 -- 172.21.15.94:0/2504389266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7000094f0 con 0x7ff71813fd70 2023-09-30T00:18:01.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff70f7fe700 1 -- 172.21.15.94:0/2504389266 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff710014480 con 0x7ff71813fd70 2023-09-30T00:18:01.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.708+0000 7ff71d816700 1 -- 172.21.15.94:0/2504389266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7180a6a40 con 0x7ff71813fd70 2023-09-30T00:18:01.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.709+0000 7ff71d816700 1 -- 172.21.15.94:0/2504389266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff718071630 con 0x7ff71813fd70 2023-09-30T00:18:01.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.710+0000 7ff70f7fe700 1 -- 172.21.15.94:0/2504389266 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ff710003020 con 0x7ff71813fd70 2023-09-30T00:18:01.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.710+0000 7ff70f7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:01.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.711+0000 7ff70f7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:01.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.711+0000 7ff70f7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:01.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.711+0000 7ff70f7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:01.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.711+0000 7ff70f7fe700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff6f8098780 0x7ff6f809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:01.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.711+0000 7ff716ffd700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff6f8098780 0x7ff6f809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:01.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.711+0000 7ff70f7fe700 1 -- 172.21.15.94:0/2504389266 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7ff7100c1690 con 0x7ff71813fd70 2023-09-30T00:18:01.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.711+0000 7ff716ffd700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff6f8098780 0x7ff6f809ac30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff708006fd0 tx=0x7ff708008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:01.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.714+0000 7ff70f7fe700 1 -- 172.21.15.94:0/2504389266 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff71008edd0 con 0x7ff71813fd70 2023-09-30T00:18:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.897+0000 7ff71d816700 1 -- 172.21.15.94:0/2504389266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7180a61c0 con 0x7ff71813fd70 2023-09-30T00:18:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.898+0000 7ff70f7fe700 1 -- 172.21.15.94:0/2504389266 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12197 (secure 0 0 0) 0x7ff710018020 con 0x7ff71813fd70 2023-09-30T00:18:01.898 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:01.898 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":19,"fsid":"3c891173-587e-4c6f-9c6a-04ec750eedad","created":"2023-09-30T00:17:34.347592+0000","modified":"2023-09-30T00:17:52.938964+0000","last_up_change":"2023-09-30T00:17:42.879328+0000","last_in_change":"2023-09-30T00:17:35.583427+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T00:17:43.481613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T00:17:45.892643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-09-30T00:17:50.220028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-09-30T00:17:50.911066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c508b402-f8ce-4461-823b-9e35fcd8d40f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":84364},{"type":"v1","addr":"172.21.15.94:6801","nonce":84364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":84364},{"type":"v1","addr":"172.21.15.94:6803","nonce":84364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":84364},{"type":"v1","addr":"172.21.15.94:6807","nonce":84364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":84364},{"type":"v1","addr":"172.21.15.94:6805","nonce":84364}]},"public_addr":"172.21.15.94:6801/84364","cluster_addr":"172.21.15.94:6803/84364","heartbeat_back_addr":"172.21.15.94:6807/84364","heartbeat_front_addr":"172.21.15.94:6805/84364","state":["exists","up"]},{"osd":1,"uuid":"e96ecdf0-bb38-4eb1-be99-54f813c12888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":86850},{"type":"v1","addr":"172.21.15.121:6817","nonce":86850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":86850},{"type":"v1","addr":"172.21.15.121:6819","nonce":86850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":86850},{"type":"v1","addr":"172.21.15.121:6823","nonce":86850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":86850},{"type":"v1","addr":"172.21.15.121:6821","nonce":86850}]},"public_addr":"172.21.15.121:6817/86850","cluster_addr":"172.21.15.121:6819/86850","heartbeat_back_addr":"172.21.15.121:6823/86850","heartbeat_front_addr":"172.21.15.121:6821/86850","state":["exists","up"]},{"osd":2,"uuid":"68700ae4-764c-4dff-9cf2-5f33132cd0d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":86820},{"type":"v1","addr":"172.21.15.121:6801","nonce":86820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":86820},{"type":"v1","addr":"172.21.15.121:6803","nonce":86820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":86820},{"type":"v1","addr":"172.21.15.121:6810","nonce":86820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":86820},{"type":"v1","addr":"172.21.15.121:6806","nonce":86820}]},"public_addr":"172.21.15.121:6801/86820","cluster_addr":"172.21.15.121:6803/86820","heartbeat_back_addr":"172.21.15.121:6810/86820","heartbeat_front_addr":"172.21.15.121:6806/86820","state":["exists","up"]},{"osd":3,"uuid":"54f17ec8-2561-42e8-bbd7-262130ffdac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":86821},{"type":"v1","addr":"172.21.15.121:6807","nonce":86821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":86821},{"type":"v1","addr":"172.21.15.121:6811","nonce":86821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":86821},{"type":"v1","addr":"172.21.15.121:6815","nonce":86821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":86821},{"type":"v1","addr":"172.21.15.121:6813","nonce":86821}]},"public_addr":"172.21.15.121:6807/86821","cluster_addr":"172.21.15.121:6811/86821","heartbeat_back_addr":"172.21.15.121:6815/86821","heartbeat_front_addr":"172.21.15.121:6813/86821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:39.951865+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.769149+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.738994+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.748197+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:18:01.900 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 10 mgrc shutdown 2023-09-30T00:18:01.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 1 -- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff6f8098780 msgr2=0x7ff6f809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff6f8098780 0x7ff6f809ac30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff708006fd0 tx=0x7ff708008040 comp rx=0 tx=0).stop 2023-09-30T00:18:01.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 1 -- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff71813fd70 msgr2=0x7ff71814ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:01.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff71813fd70 0x7ff71814ff30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff710009f20 tx=0x7ff71000b040 comp rx=0 tx=0).stop 2023-09-30T00:18:01.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 1 -- 172.21.15.94:0/2504389266 shutdown_connections 2023-09-30T00:18:01.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff6f8098780 0x7ff6f809ac30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff71813fd70 0x7ff71814ff30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff718070c50 0x7ff7180a88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 1 --2- 172.21.15.94:0/2504389266 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff7180b7160 0x7ff7180a6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:01.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 1 -- 172.21.15.94:0/2504389266 >> 172.21.15.94:0/2504389266 conn(0x7ff71806a090 msgr2=0x7ff7180b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:01.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 1 -- 172.21.15.94:0/2504389266 shutdown_connections 2023-09-30T00:18:01.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:01.900+0000 7ff71d816700 1 -- 172.21.15.94:0/2504389266 wait complete. 2023-09-30T00:18:01.912 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T00:18:01.913 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T00:18:01.913 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T00:18:01.914 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T00:18:02.035 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.035+0000 7f872063c700 1 Processor -- start 2023-09-30T00:18:02.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.035+0000 7fe3e0375700 1 Processor -- start 2023-09-30T00:18:02.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.035+0000 7fe3e0375700 1 -- start start 2023-09-30T00:18:02.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.035+0000 7f872063c700 1 -- start start 2023-09-30T00:18:02.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.035+0000 7f5f3911d700 1 Processor -- start 2023-09-30T00:18:02.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.035+0000 7f5f3911d700 1 -- start start 2023-09-30T00:18:02.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.035+0000 7fcd10bd3700 1 Processor -- start 2023-09-30T00:18:02.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.035+0000 7fcd10bd3700 1 -- start start 2023-09-30T00:18:02.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f5f3911d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5f340a0e40 0x7f5f3414bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f5f3911d700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f340b7270 con 0x7f5f340a2020 2023-09-30T00:18:02.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f5f3911d700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f340a18d0 con 0x7f5f340a23f0 2023-09-30T00:18:02.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f5f3911d700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f34140540 con 0x7f5f340a0e40 2023-09-30T00:18:02.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f5f33fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5f340a2020 0x7f5f340b5e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50220/0 (socket says 172.21.15.94:50220) 2023-09-30T00:18:02.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f5f33fff700 1 -- 172.21.15.94:0/2167384449 learned_addr learned my addr 172.21.15.94:0/2167384449 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:02.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fe3e0375700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe3d809de30 0x7fe3d8140b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fe3e0375700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d8141180 con 0x7fe3d806eab0 2023-09-30T00:18:02.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fe3e0375700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d8141440 con 0x7fe3d80a2ef0 2023-09-30T00:18:02.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fe3e0375700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d8141700 con 0x7fe3d809de30 2023-09-30T00:18:02.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fe3de111700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe3d809de30 0x7fe3d8140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fe3de111700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe3d809de30 0x7fe3d8140b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50652/0 (socket says 172.21.15.94:50652) 2023-09-30T00:18:02.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fe3de111700 1 -- 172.21.15.94:0/25365963 learned_addr learned my addr 172.21.15.94:0/25365963 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f872063c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f871809de30 0x7f8718140b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f872063c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f871806eab0 0x7f871809f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.041 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fcd10bd3700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcd0c140bc0 0x7fcd0c140fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.041 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fcd10bd3700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcd0c09fa60 0x7fcd0c09fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.041 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fcd10bd3700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0c0a0400 con 0x7fcd0c06e0a0 2023-09-30T00:18:02.041 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fcd10bd3700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd0c06e470 con 0x7fcd0c09fa60 2023-09-30T00:18:02.041 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fcd10bd3700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd0c140690 con 0x7fcd0c140bc0 2023-09-30T00:18:02.041 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030549573 0 0) 0x7f5f340b7270 con 0x7f5f340a2020 2023-09-30T00:18:02.041 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f1c003570 con 0x7f5f340a2020 2023-09-30T00:18:02.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f872063c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871809fdc0 con 0x7f87180a2ef0 2023-09-30T00:18:02.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f872063c700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8718141180 con 0x7f871806eab0 2023-09-30T00:18:02.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f872063c700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8718141440 con 0x7f871809de30 2023-09-30T00:18:02.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fcd0affd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcd0c140bc0 0x7fcd0c140fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fe3de111700 1 -- 172.21.15.94:0/25365963 >> v1:172.21.15.121:6789/0 conn(0x7fe3d80a2ef0 legacy=0x7fe3d8147e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:02.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fe3de111700 1 -- 172.21.15.94:0/25365963 >> v1:172.21.15.94:6789/0 conn(0x7fe3d806eab0 legacy=0x7fe3d809f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:18:02.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fcd0affd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcd0c140bc0 0x7fcd0c140fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.94:44610/0 (socket says 172.21.15.94:44610) 2023-09-30T00:18:02.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fcd0affd700 1 -- 172.21.15.94:0/2196938588 learned_addr learned my addr 172.21.15.94:0/2196938588 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7fe3de111700 1 -- 172.21.15.94:0/25365963 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d809e200 con 0x7fe3d809de30 2023-09-30T00:18:02.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f5f3891c700 1 --2- 172.21.15.94:0/2167384449 >> v2:172.21.15.94:3300/0 conn(0x7f5f340a0e40 0x7f5f3414bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f871dbd7700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f871806eab0 0x7f871809f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f871e3d8700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f871809de30 0x7f8718140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f871dbd7700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f871806eab0 0x7f871809f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50656/0 (socket says 172.21.15.94:50656) 2023-09-30T00:18:02.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f871dbd7700 1 -- 172.21.15.94:0/1990486095 learned_addr learned my addr 172.21.15.94:0/1990486095 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.036+0000 7f871e3d8700 1 --2- 172.21.15.94:0/1990486095 >> v2:172.21.15.121:3300/0 conn(0x7f871809de30 0x7f8718140b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.94:44620/0 (socket says 172.21.15.94:44620) 2023-09-30T00:18:02.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fe3de111700 1 --2- 172.21.15.94:0/25365963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3d809de30 0x7fe3d8140b00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400bc50 tx=0x7fe3d400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c93da880b90fd58e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fcd0bfff700 1 --2- 172.21.15.94:0/2196938588 >> v2:172.21.15.94:3300/0 conn(0x7fcd0c09fa60 0x7fcd0c09fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fcd0affd700 1 -- 172.21.15.94:0/2196938588 >> v2:172.21.15.94:3300/0 conn(0x7fcd0c09fa60 msgr2=0x7fcd0c09fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:02.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fcd0affd700 1 --2- 172.21.15.94:0/2196938588 >> v2:172.21.15.94:3300/0 conn(0x7fcd0c09fa60 0x7fcd0c09fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f871dbd7700 1 -- 172.21.15.94:0/1990486095 >> v2:172.21.15.121:3300/0 conn(0x7f871809de30 msgr2=0x7f8718140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:02.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fcd0affd700 1 -- 172.21.15.94:0/2196938588 >> v1:172.21.15.94:6789/0 conn(0x7fcd0c06e0a0 legacy=0x7fcd0c09e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:02.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fcd0affd700 1 -- 172.21.15.94:0/2196938588 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0c140930 con 0x7fcd0c140bc0 2023-09-30T00:18:02.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f871dbd7700 1 --2- 172.21.15.94:0/1990486095 >> v2:172.21.15.121:3300/0 conn(0x7f871809de30 0x7f8718140b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f871dbd7700 1 -- 172.21.15.94:0/1990486095 >> v1:172.21.15.94:6789/0 conn(0x7f87180a2ef0 legacy=0x7f87180a4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f871dbd7700 1 -- 172.21.15.94:0/1990486095 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8718141700 con 0x7f871806eab0 2023-09-30T00:18:02.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f871e3d8700 1 --2- 172.21.15.94:0/1990486095 >> v2:172.21.15.121:3300/0 conn(0x7f871809de30 0x7f8718140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:02.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fe3dd10f700 1 -- 172.21.15.94:0/25365963 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d400be60 con 0x7fe3d809de30 2023-09-30T00:18:02.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fe3dd10f700 1 -- 172.21.15.94:0/25365963 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d400de10 con 0x7fe3d809de30 2023-09-30T00:18:02.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fe3dd10f700 1 -- 172.21.15.94:0/25365963 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d40106a0 con 0x7fe3d809de30 2023-09-30T00:18:02.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fcd0affd700 1 --2- 172.21.15.94:0/2196938588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0c140bc0 0x7fcd0c140fc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c06eb50 tx=0x7fcd00009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbcecf385dd541ca server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654801686 0 0) 0x7f5f340a18d0 con 0x7f5f340a23f0 2023-09-30T00:18:02.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f340b7270 con 0x7f5f340a23f0 2023-09-30T00:18:02.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1960600673 0 0) 0x7f5f1c003570 con 0x7f5f340a2020 2023-09-30T00:18:02.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 >> v2:172.21.15.94:3300/0 conn(0x7f5f340a0e40 msgr2=0x7f5f3414bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:02.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 --2- 172.21.15.94:0/2167384449 >> v2:172.21.15.94:3300/0 conn(0x7f5f340a0e40 0x7f5f3414bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 >> v1:172.21.15.121:6789/0 conn(0x7f5f340a23f0 legacy=0x7f5f340a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f341407a0 con 0x7f5f340a2020 2023-09-30T00:18:02.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3153706020 0 0) 0x7f5f24001470 con 0x7f5f340a2020 2023-09-30T00:18:02.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 >> v1:172.21.15.94:6789/0 conn(0x7f5f340a2020 legacy=0x7f5f340b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 --2- 172.21.15.94:0/2167384449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f1c003c40 0x7f5f1c004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 --2- 172.21.15.94:0/2167384449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f1c004810 0x7f5f1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 --2- 172.21.15.94:0/2167384449 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f1c005210 0x7f5f1c014270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f34140540 con 0x7f5f1c003c40 2023-09-30T00:18:02.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c014940 con 0x7f5f1c004810 2023-09-30T00:18:02.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c005a30 con 0x7f5f1c005210 2023-09-30T00:18:02.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fcd0a7fc700 1 -- 172.21.15.94:0/2196938588 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0000b820 con 0x7fcd0c140bc0 2023-09-30T00:18:02.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f33fff700 1 --2- 172.21.15.94:0/2167384449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f1c003c40 0x7f5f1c004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fcd0a7fc700 1 -- 172.21.15.94:0/2196938588 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0000b980 con 0x7fcd0c140bc0 2023-09-30T00:18:02.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7fcd0a7fc700 1 -- 172.21.15.94:0/2196938588 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0000bd20 con 0x7fcd0c140bc0 2023-09-30T00:18:02.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f871dbd7700 1 --2- 172.21.15.94:0/1990486095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f871806eab0 0x7f871809f880 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f87180695b0 tx=0x7f870800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d1ca430f24ddedc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f337fe700 1 --2- 172.21.15.94:0/2167384449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f1c004810 0x7f5f1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f3891c700 1 --2- 172.21.15.94:0/2167384449 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f1c005210 0x7f5f1c014270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f871d3d6700 1 -- 172.21.15.94:0/1990486095 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870800dce0 con 0x7f871806eab0 2023-09-30T00:18:02.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f33fff700 1 -- 172.21.15.94:0/2167384449 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f1c005210 msgr2=0x7f5f1c014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:02.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f33fff700 1 --2- 172.21.15.94:0/2167384449 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f1c005210 0x7f5f1c014270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f33fff700 1 -- 172.21.15.94:0/2167384449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f1c004810 msgr2=0x7f5f1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:02.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f33fff700 1 --2- 172.21.15.94:0/2167384449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f1c004810 0x7f5f1c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f33fff700 1 -- 172.21.15.94:0/2167384449 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c003570 con 0x7f5f1c003c40 2023-09-30T00:18:02.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f3891c700 1 --2- 172.21.15.94:0/2167384449 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f1c005210 0x7f5f1c014270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:02.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f5f3891c700 1 -- 172.21.15.94:0/2167384449 reap_dead start 2023-09-30T00:18:02.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f871d3d6700 1 -- 172.21.15.94:0/1990486095 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870800de40 con 0x7f871806eab0 2023-09-30T00:18:02.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.037+0000 7f871d3d6700 1 -- 172.21.15.94:0/1990486095 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87080196d0 con 0x7f871806eab0 2023-09-30T00:18:02.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7f5f33fff700 1 --2- 172.21.15.94:0/2167384449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f1c003c40 0x7f5f1c004030 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400c240 tx=0x7f5f24001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e11acea5cac6cd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2400cc00 con 0x7f5f1c003c40 2023-09-30T00:18:02.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7fe3e0375700 1 -- 172.21.15.94:0/25365963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3d809de30 msgr2=0x7fe3d8140b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7fe3e0375700 1 --2- 172.21.15.94:0/25365963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3d809de30 0x7fe3d8140b00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400bc50 tx=0x7fe3d400b4f0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2400cda0 con 0x7f5f1c003c40 2023-09-30T00:18:02.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7f5f32ffd700 1 -- 172.21.15.94:0/2167384449 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24005b70 con 0x7f5f1c003c40 2023-09-30T00:18:02.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7fe3e0375700 1 -- 172.21.15.94:0/25365963 shutdown_connections 2023-09-30T00:18:02.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7fe3e0375700 1 --2- 172.21.15.94:0/25365963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3d809de30 0x7fe3d8140b00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7fe3e0375700 1 -- 172.21.15.94:0/25365963 >> 172.21.15.94:0/25365963 conn(0x7fe3d80b3520 msgr2=0x7fe3d8067620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7fe3e0375700 1 -- 172.21.15.94:0/25365963 shutdown_connections 2023-09-30T00:18:02.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7fe3e0375700 1 -- 172.21.15.94:0/25365963 wait complete. 2023-09-30T00:18:02.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7f872063c700 1 -- 172.21.15.94:0/1990486095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f871806eab0 msgr2=0x7f871809f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7f872063c700 1 --2- 172.21.15.94:0/1990486095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f871806eab0 0x7f871809f880 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f87180695b0 tx=0x7f870800b4f0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7f872063c700 1 -- 172.21.15.94:0/1990486095 shutdown_connections 2023-09-30T00:18:02.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7f872063c700 1 --2- 172.21.15.94:0/1990486095 >> v2:172.21.15.121:3300/0 conn(0x7f871809de30 0x7f8718140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7f872063c700 1 --2- 172.21.15.94:0/1990486095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f871806eab0 0x7f871809f880 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7f872063c700 1 -- 172.21.15.94:0/1990486095 >> 172.21.15.94:0/1990486095 conn(0x7f87180b3520 msgr2=0x7f8718067620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7f872063c700 1 -- 172.21.15.94:0/1990486095 shutdown_connections 2023-09-30T00:18:02.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.038+0000 7f872063c700 1 -- 172.21.15.94:0/1990486095 wait complete. 2023-09-30T00:18:02.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2196938588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0c140bc0 msgr2=0x7fcd0c140fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7fcd10bd3700 1 --2- 172.21.15.94:0/2196938588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0c140bc0 0x7fcd0c140fc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c06eb50 tx=0x7fcd00009250 comp rx=0 tx=0).stop 2023-09-30T00:18:02.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7f5f3911d700 1 -- 172.21.15.94:0/2167384449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f1c003c40 msgr2=0x7f5f1c004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7f5f3911d700 1 --2- 172.21.15.94:0/2167384449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f1c003c40 0x7f5f1c004030 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400c240 tx=0x7f5f24001d40 comp rx=0 tx=0).stop 2023-09-30T00:18:02.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7f5f3911d700 1 -- 172.21.15.94:0/2167384449 shutdown_connections 2023-09-30T00:18:02.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7f5f3911d700 1 --2- 172.21.15.94:0/2167384449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f1c003c40 0x7f5f1c004030 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7f5f3911d700 1 -- 172.21.15.94:0/2167384449 >> 172.21.15.94:0/2167384449 conn(0x7f5f34070dc0 msgr2=0x7f5f34067bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2196938588 shutdown_connections 2023-09-30T00:18:02.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7fcd10bd3700 1 --2- 172.21.15.94:0/2196938588 >> v2:172.21.15.94:3300/0 conn(0x7fcd0c09fa60 0x7fcd0c09fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7fcd10bd3700 1 --2- 172.21.15.94:0/2196938588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0c140bc0 0x7fcd0c140fc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2196938588 >> 172.21.15.94:0/2196938588 conn(0x7fcd0c0b43f0 msgr2=0x7fcd0c06c760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2196938588 shutdown_connections 2023-09-30T00:18:02.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2196938588 wait complete. 2023-09-30T00:18:02.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7f5f3911d700 1 -- 172.21.15.94:0/2167384449 shutdown_connections 2023-09-30T00:18:02.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7f5f3911d700 1 -- 172.21.15.94:0/2167384449 wait complete. 2023-09-30T00:18:02.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7fe3e0375700 1 Processor -- start 2023-09-30T00:18:02.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7fe3e0375700 1 -- start start 2023-09-30T00:18:02.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7f872063c700 1 Processor -- start 2023-09-30T00:18:02.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.039+0000 7f872063c700 1 -- start start 2023-09-30T00:18:02.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7f5f3911d700 1 Processor -- start 2023-09-30T00:18:02.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7fe3e0375700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe3d806eab0 0x7fe3d806b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7fe3e0375700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3d809de30 0x7fe3d813f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7fe3e0375700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3d80a2ef0 0x7fe3d8096740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7fe3e0375700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d806c0c0 con 0x7fe3d809de30 2023-09-30T00:18:02.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7fe3e0375700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d806c360 con 0x7fe3d80a2ef0 2023-09-30T00:18:02.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7fe3e0375700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d806c620 con 0x7fe3d806eab0 2023-09-30T00:18:02.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7fe3de111700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe3d806eab0 0x7fe3d806b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7fe3de111700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe3d806eab0 0x7fe3d806b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39154/0 (socket says 172.21.15.94:39154) 2023-09-30T00:18:02.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7fe3de111700 1 -- 172.21.15.94:0/2435658022 learned_addr learned my addr 172.21.15.94:0/2435658022 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7fe3dd910700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3d809de30 0x7fe3d813f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7fcd10bd3700 1 Processor -- start 2023-09-30T00:18:02.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7f5f3911d700 1 -- start start 2023-09-30T00:18:02.064 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7fcd10bd3700 1 -- start start 2023-09-30T00:18:02.064 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7f872063c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f871806eab0 0x7f8718096b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.065 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fe3de111700 1 -- 172.21.15.94:0/2435658022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3d80a2ef0 msgr2=0x7fe3d8096740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.065 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fe3de111700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3d80a2ef0 0x7fe3d8096740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.065 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fe3de111700 1 -- 172.21.15.94:0/2435658022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3d809de30 msgr2=0x7fe3d813f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.065 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fe3de111700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3d809de30 0x7fe3d813f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.065 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fe3de111700 1 -- 172.21.15.94:0/2435658022 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d400b790 con 0x7fe3d806eab0 2023-09-30T00:18:02.065 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fe3dd910700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3d809de30 0x7fe3d813f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:02.065 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fe3de111700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe3d806eab0 0x7fe3d806b870 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400b760 tx=0x7fe3d401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.066 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3911d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f1c003c40 0x7f5f34139330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.066 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3911d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f340a3a10 0x7f5f3406aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.066 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3911d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f34139870 0x7f5f3406b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.066 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3911d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340a6000 con 0x7f5f1c003c40 2023-09-30T00:18:02.066 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3911d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3414cc80 con 0x7f5f340a3a10 2023-09-30T00:18:02.066 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3911d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3414cf40 con 0x7f5f34139870 2023-09-30T00:18:02.066 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7f872063c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f871809de30 0x7f871813f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.067 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7f872063c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f87180a2ef0 0x7f871813a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.067 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7f872063c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871814f8b0 con 0x7f871809de30 2023-09-30T00:18:02.067 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7f872063c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8718095e40 con 0x7f871806eab0 2023-09-30T00:18:02.067 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.040+0000 7f872063c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871814fae0 con 0x7f87180a2ef0 2023-09-30T00:18:02.067 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f871e3d8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f871806eab0 0x7f8718096b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.067 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f871ebd9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f87180a2ef0 0x7f871813a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.067 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f871e3d8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f871806eab0 0x7f8718096b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.94:44634/0 (socket says 172.21.15.94:44634) 2023-09-30T00:18:02.067 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f871e3d8700 1 -- 172.21.15.94:0/1344320201 learned_addr learned my addr 172.21.15.94:0/1344320201 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.067 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f871dbd7700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f871809de30 0x7f871813f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.068 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3891c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f34139870 0x7f5f3406b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.068 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f337fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f340a3a10 0x7f5f3406aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.068 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f337fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f340a3a10 0x7f5f3406aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.94:44638/0 (socket says 172.21.15.94:44638) 2023-09-30T00:18:02.068 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3891c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f34139870 0x7f5f3406b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39182/0 (socket says 172.21.15.94:39182) 2023-09-30T00:18:02.068 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f337fe700 1 -- 172.21.15.94:0/802123000 learned_addr learned my addr 172.21.15.94:0/802123000 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.068 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f33fff700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f1c003c40 0x7f5f34139330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.068 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f871e3d8700 1 -- 172.21.15.94:0/1344320201 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f87180a2ef0 msgr2=0x7f871813a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.068 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f871e3d8700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f87180a2ef0 0x7f871813a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.069 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f871e3d8700 1 -- 172.21.15.94:0/1344320201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f871809de30 msgr2=0x7f871813f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.069 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f871e3d8700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f871809de30 0x7f871813f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.069 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f871e3d8700 1 -- 172.21.15.94:0/1344320201 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870800b790 con 0x7f871806eab0 2023-09-30T00:18:02.069 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f871dbd7700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f871809de30 0x7f871813f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:02.069 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f871e3d8700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f871806eab0 0x7f8718096b10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f871400e900 tx=0x7f8714008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.069 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3891c700 1 -- 172.21.15.94:0/802123000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f340a3a10 msgr2=0x7f5f3406aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.069 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3891c700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f340a3a10 0x7f5f3406aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.070 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fcd10bd3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd0c06e0a0 0x7fcd0c067c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.070 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fcd10bd3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd0c09fa60 0x7fcd0c0692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.070 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3891c700 1 -- 172.21.15.94:0/802123000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f1c003c40 msgr2=0x7f5f34139330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.070 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3891c700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f1c003c40 0x7f5f34139330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.071 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fcd10bd3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0c140bc0 0x7fcd0c13a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.071 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fcd10bd3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c068e30 con 0x7fcd0c06e0a0 2023-09-30T00:18:02.071 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fcd10bd3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c141e10 con 0x7fcd0c140bc0 2023-09-30T00:18:02.071 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fcd10bd3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c1420d0 con 0x7fcd0c09fa60 2023-09-30T00:18:02.071 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fcd0b7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd0c06e0a0 0x7fcd0c067c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.071 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f3891c700 1 -- 172.21.15.94:0/802123000 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f340a5d50 con 0x7f5f34139870 2023-09-30T00:18:02.071 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fcd0b7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd0c06e0a0 0x7fcd0c067c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50716/0 (socket says 172.21.15.94:50716) 2023-09-30T00:18:02.072 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fcd0b7fe700 1 -- 172.21.15.94:0/2924068270 learned_addr learned my addr 172.21.15.94:0/2924068270 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.072 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fe3ce7fc700 1 -- 172.21.15.94:0/2435658022 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4018880 con 0x7fe3d806eab0 2023-09-30T00:18:02.072 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f33fff700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f1c003c40 0x7f5f34139330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:02.072 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fe3ce7fc700 1 -- 172.21.15.94:0/2435658022 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d40189e0 con 0x7fe3d806eab0 2023-09-30T00:18:02.072 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7fe3ce7fc700 1 -- 172.21.15.94:0/2435658022 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4018d60 con 0x7fe3d806eab0 2023-09-30T00:18:02.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7f5f337fe700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f340a3a10 0x7f5f3406aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:02.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7fe3e0375700 1 -- 172.21.15.94:0/2435658022 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d806bdb0 con 0x7fe3d806eab0 2023-09-30T00:18:02.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.041+0000 7fcd0bfff700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0c140bc0 0x7fcd0c13a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7fcd0affd700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd0c09fa60 0x7fcd0c0692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7f870e7fc700 1 -- 172.21.15.94:0/1344320201 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8714013410 con 0x7f871806eab0 2023-09-30T00:18:02.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7f870e7fc700 1 -- 172.21.15.94:0/1344320201 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8714013570 con 0x7f871806eab0 2023-09-30T00:18:02.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7f870e7fc700 1 -- 172.21.15.94:0/1344320201 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8714013910 con 0x7f871806eab0 2023-09-30T00:18:02.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7fe3e0375700 1 -- 172.21.15.94:0/2435658022 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d8096c80 con 0x7fe3d806eab0 2023-09-30T00:18:02.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7f5f3891c700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f34139870 0x7f5f3406b400 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5f340a27c0 tx=0x7f5f2c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.075 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7f872063c700 1 -- 172.21.15.94:0/1344320201 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871814fda0 con 0x7f871806eab0 2023-09-30T00:18:02.075 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7f872063c700 1 -- 172.21.15.94:0/1344320201 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8718140530 con 0x7f871806eab0 2023-09-30T00:18:02.075 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7fcd0bfff700 1 -- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd0c09fa60 msgr2=0x7fcd0c0692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.075 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7fcd0bfff700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd0c09fa60 0x7fcd0c0692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.075 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7fcd0bfff700 1 -- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd0c06e0a0 msgr2=0x7fcd0c067c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7fcd0bfff700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd0c06e0a0 0x7fcd0c067c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7fcd0bfff700 1 -- 172.21.15.94:0/2924068270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd000094f0 con 0x7fcd0c140bc0 2023-09-30T00:18:02.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7fcd0affd700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd0c09fa60 0x7fcd0c0692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:02.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7fcd0b7fe700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd0c06e0a0 0x7fcd0c067c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:02.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.042+0000 7fcd0bfff700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0c140bc0 0x7fcd0c13a480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcd04007ed0 tx=0x7fcd04009850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.043+0000 7fe3e0375700 1 -- 172.21.15.94:0/2435658022 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe3d8071630 con 0x7fe3d806eab0 2023-09-30T00:18:02.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.043+0000 7fccf37fe700 1 -- 172.21.15.94:0/2924068270 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0400eb90 con 0x7fcd0c140bc0 2023-09-30T00:18:02.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.043+0000 7f5f1bfff700 1 -- 172.21.15.94:0/802123000 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c00e050 con 0x7f5f34139870 2023-09-30T00:18:02.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.043+0000 7f5f1bfff700 1 -- 172.21.15.94:0/802123000 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c00f040 con 0x7f5f34139870 2023-09-30T00:18:02.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.043+0000 7f5f3911d700 1 -- 172.21.15.94:0/802123000 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3414bf80 con 0x7f5f34139870 2023-09-30T00:18:02.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.043+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2924068270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0c069820 con 0x7fcd0c140bc0 2023-09-30T00:18:02.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.043+0000 7fccf37fe700 1 -- 172.21.15.94:0/2924068270 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0400ecf0 con 0x7fcd0c140bc0 2023-09-30T00:18:02.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.043+0000 7fccf37fe700 1 -- 172.21.15.94:0/2924068270 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd04014670 con 0x7fcd0c140bc0 2023-09-30T00:18:02.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.043+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2924068270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd0c06a500 con 0x7fcd0c140bc0 2023-09-30T00:18:02.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.043+0000 7f5f1bfff700 1 -- 172.21.15.94:0/802123000 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c010e30 con 0x7f5f34139870 2023-09-30T00:18:02.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.043+0000 7f5f3911d700 1 -- 172.21.15.94:0/802123000 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f3414c470 con 0x7f5f34139870 2023-09-30T00:18:02.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.044+0000 7f5f3911d700 1 -- 172.21.15.94:0/802123000 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5f3404c380 con 0x7f5f34139870 2023-09-30T00:18:02.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.043+0000 7f872063c700 1 -- 172.21.15.94:0/1344320201 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8700000f10 con 0x7f871806eab0 2023-09-30T00:18:02.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.044+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2924068270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcd0c06a6f0 con 0x7fcd0c140bc0 2023-09-30T00:18:02.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.044+0000 7fe3ce7fc700 1 -- 172.21.15.94:0/2435658022 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe3d40233d0 con 0x7fe3d806eab0 2023-09-30T00:18:02.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.044+0000 7fe3ce7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:02.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7fe3ce7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:02.080 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7fe3ce7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.080 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7fe3ce7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.080 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7fe3ce7fc700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe3bc098780 0x7fe3bc09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.080 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7f870e7fc700 1 -- 172.21.15.94:0/1344320201 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f8714013d00 con 0x7f871806eab0 2023-09-30T00:18:02.080 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7f870e7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:02.080 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7fccf37fe700 1 -- 172.21.15.94:0/2924068270 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fcd04003020 con 0x7fcd0c140bc0 2023-09-30T00:18:02.080 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7fccf37fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:02.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7fe3ce7fc700 1 -- 172.21.15.94:0/2435658022 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fe3d4014070 con 0x7fe3d806eab0 2023-09-30T00:18:02.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7fe3ce7fc700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.121:6816/86850,v1:172.21.15.121:6817/86850] conn(0x7fe3bc0a02e0 0x7fe3bc0a2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7fe3ce7fc700 1 -- 172.21.15.94:0/2435658022 --> [v2:172.21.15.121:6816/86850,v1:172.21.15.121:6817/86850] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3bc0a2de0 con 0x7fe3bc0a02e0 2023-09-30T00:18:02.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7fe3ce7fc700 1 -- 172.21.15.94:0/2435658022 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3d40c4db0 con 0x7fe3d806eab0 2023-09-30T00:18:02.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7f870e7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:02.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7f870e7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7f870e7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7f870e7fc700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f86fc0987b0 0x7f86fc09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7fe3de912700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.121:6816/86850,v1:172.21.15.121:6817/86850] conn(0x7fe3bc0a02e0 0x7fe3bc0a2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.045+0000 7f871dbd7700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f86fc0987b0 0x7f86fc09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7f870e7fc700 1 -- 172.21.15.94:0/1344320201 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f87140c00e0 con 0x7f871806eab0 2023-09-30T00:18:02.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fccf37fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:02.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fccf37fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fccf37fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fccf37fe700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fccf8098780 0x7fccf809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7f870e7fc700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.121:6805/86821,v1:172.21.15.121:6807/86821] conn(0x7f86fc0a0310 0x7f86fc0a2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7f870e7fc700 1 -- 172.21.15.94:0/1344320201 --> [v2:172.21.15.121:6805/86821,v1:172.21.15.121:6807/86821] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86fc0a2e10 con 0x7f86fc0a0310 2023-09-30T00:18:02.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7f870e7fc700 1 -- 172.21.15.94:0/1344320201 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87140c26f0 con 0x7f871806eab0 2023-09-30T00:18:02.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fcd0b7fe700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fccf8098780 0x7fccf809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fccf37fe700 1 -- 172.21.15.94:0/2924068270 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fcd040c1920 con 0x7fcd0c140bc0 2023-09-30T00:18:02.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7f871dbd7700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f86fc0987b0 0x7f86fc09ac60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f87180695b0 tx=0x7f870801b000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7f871ebd9700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.121:6805/86821,v1:172.21.15.121:6807/86821] conn(0x7f86fc0a0310 0x7f86fc0a2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fccf37fe700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:6800/84364,v1:172.21.15.94:6801/84364] conn(0x7fccf80a02e0 0x7fccf80a2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fccf37fe700 1 -- 172.21.15.94:0/2924068270 --> [v2:172.21.15.94:6800/84364,v1:172.21.15.94:6801/84364] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fccf80a2de0 con 0x7fccf80a02e0 2023-09-30T00:18:02.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fccf37fe700 1 -- 172.21.15.94:0/2924068270 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd040c1c10 con 0x7fcd0c140bc0 2023-09-30T00:18:02.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7f5f1bfff700 1 -- 172.21.15.94:0/802123000 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5f2c015650 con 0x7f5f34139870 2023-09-30T00:18:02.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7f5f1bfff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:02.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fe3de912700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.121:6816/86850,v1:172.21.15.121:6817/86850] conn(0x7fe3bc0a02e0 0x7fe3bc0a2710 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.086 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fe3dd910700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe3bc098780 0x7fe3bc09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7f871ebd9700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.121:6805/86821,v1:172.21.15.121:6807/86821] conn(0x7f86fc0a0310 0x7f86fc0a2740 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.087 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7fe3dd910700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe3bc098780 0x7fe3bc09ac30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe3d813cb50 tx=0x7fe3c8009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7f5f1bfff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:02.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7f5f1bfff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7f5f1bfff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7f5f1bfff700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5f20098850 0x7f5f2009ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7f5f33fff700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5f20098850 0x7f5f2009ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7f5f1bfff700 1 -- 172.21.15.94:0/802123000 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f5f2c0c1990 con 0x7f5f34139870 2023-09-30T00:18:02.088 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7fe3ce7fc700 1 -- 172.21.15.94:0/2435658022 <== osd.1 v2:172.21.15.121:6816/86850 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fe3bc0a2de0 con 0x7fe3bc0a02e0 2023-09-30T00:18:02.089 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fcd0affd700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:6800/84364,v1:172.21.15.94:6801/84364] conn(0x7fccf80a02e0 0x7fccf80a2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.089 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fcd0b7fe700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fccf8098780 0x7fccf809ac30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fccfc007fd0 tx=0x7fccfc007f60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.089 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.046+0000 7fcd0affd700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:6800/84364,v1:172.21.15.94:6801/84364] conn(0x7fccf80a02e0 0x7fccf80a2710 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.089 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7f5f1bfff700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.121:6800/86820,v1:172.21.15.121:6801/86820] conn(0x7f5f200a03b0 0x7f5f200a27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.089 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7f5f1bfff700 1 -- 172.21.15.94:0/802123000 --> [v2:172.21.15.121:6800/86820,v1:172.21.15.121:6801/86820] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f200a2eb0 con 0x7f5f200a03b0 2023-09-30T00:18:02.089 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7f5f1bfff700 1 -- 172.21.15.94:0/802123000 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f2c0c1d60 con 0x7f5f34139870 2023-09-30T00:18:02.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7f870e7fc700 1 -- 172.21.15.94:0/1344320201 <== osd.3 v2:172.21.15.121:6805/86821 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f86fc0a2e10 con 0x7f86fc0a0310 2023-09-30T00:18:02.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7f5f33fff700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5f20098850 0x7f5f2009ad00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400c240 tx=0x7f5f24015000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.090 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7f5f337fe700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.121:6800/86820,v1:172.21.15.121:6801/86820] conn(0x7f5f200a03b0 0x7f5f200a27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.047+0000 7fccf37fe700 1 -- 172.21.15.94:0/2924068270 <== osd.0 v2:172.21.15.94:6800/84364 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fccf80a2de0 con 0x7fccf80a02e0 2023-09-30T00:18:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.048+0000 7f5f337fe700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.121:6800/86820,v1:172.21.15.121:6801/86820] conn(0x7f5f200a03b0 0x7f5f200a27e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.048+0000 7f5f1bfff700 1 -- 172.21.15.94:0/802123000 <== osd.2 v2:172.21.15.121:6800/86820 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f5f200a2eb0 con 0x7f5f200a03b0 2023-09-30T00:18:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.062+0000 7fe3e0375700 1 -- 172.21.15.94:0/2435658022 --> [v2:172.21.15.121:6816/86850,v1:172.21.15.121:6817/86850] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe3d806cc90 con 0x7fe3bc0a02e0 2023-09-30T00:18:02.092 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.062+0000 7f872063c700 1 -- 172.21.15.94:0/1344320201 --> [v2:172.21.15.121:6805/86821,v1:172.21.15.121:6807/86821] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8700002bc0 con 0x7f86fc0a0310 2023-09-30T00:18:02.092 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.062+0000 7f870e7fc700 1 -- 172.21.15.94:0/1344320201 <== osd.3 v2:172.21.15.121:6805/86821 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8700002bc0 con 0x7f86fc0a0310 2023-09-30T00:18:02.092 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fe3ce7fc700 1 -- 172.21.15.94:0/2435658022 <== osd.1 v2:172.21.15.121:6816/86850 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe3d806cc90 con 0x7fe3bc0a02e0 2023-09-30T00:18:02.093 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2924068270 --> [v2:172.21.15.94:6800/84364,v1:172.21.15.94:6801/84364] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcd0c13d5a0 con 0x7fccf80a02e0 2023-09-30T00:18:02.093 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7f872063c700 1 -- 172.21.15.94:0/1344320201 >> [v2:172.21.15.121:6805/86821,v1:172.21.15.121:6807/86821] conn(0x7f86fc0a0310 msgr2=0x7f86fc0a2740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.093 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7f872063c700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.121:6805/86821,v1:172.21.15.121:6807/86821] conn(0x7f86fc0a0310 0x7f86fc0a2740 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.093 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fe3e0375700 1 -- 172.21.15.94:0/2435658022 >> [v2:172.21.15.121:6816/86850,v1:172.21.15.121:6817/86850] conn(0x7fe3bc0a02e0 msgr2=0x7fe3bc0a2710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.093 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fe3e0375700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.121:6816/86850,v1:172.21.15.121:6817/86850] conn(0x7fe3bc0a02e0 0x7fe3bc0a2710 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.094 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7f872063c700 10 mgrc shutdown 2023-09-30T00:18:02.094 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fe3e0375700 10 mgrc shutdown 2023-09-30T00:18:02.094 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fe3e0375700 1 -- 172.21.15.94:0/2435658022 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe3bc098780 msgr2=0x7fe3bc09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.094 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fe3e0375700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe3bc098780 0x7fe3bc09ac30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe3d813cb50 tx=0x7fe3c8009060 comp rx=0 tx=0).stop 2023-09-30T00:18:02.094 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fe3e0375700 1 -- 172.21.15.94:0/2435658022 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe3d806eab0 msgr2=0x7fe3d806b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.095 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fe3e0375700 1 --2- 172.21.15.94:0/2435658022 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe3d806eab0 0x7fe3d806b870 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400b760 tx=0x7fe3d401a000 comp rx=0 tx=0).stop 2023-09-30T00:18:02.095 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7f872063c700 1 -- 172.21.15.94:0/1344320201 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f86fc0987b0 msgr2=0x7f86fc09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.095 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7f872063c700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f86fc0987b0 0x7f86fc09ac60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f87180695b0 tx=0x7f870801b000 comp rx=0 tx=0).stop 2023-09-30T00:18:02.095 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7f872063c700 1 -- 172.21.15.94:0/1344320201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f871806eab0 msgr2=0x7f8718096b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.095 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7f872063c700 1 --2- 172.21.15.94:0/1344320201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f871806eab0 0x7f8718096b10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f871400e900 tx=0x7f8714008040 comp rx=0 tx=0).stop 2023-09-30T00:18:02.095 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fe3de912700 1 -- 172.21.15.94:0/2435658022 reap_dead start 2023-09-30T00:18:02.096 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7f871ebd9700 1 -- 172.21.15.94:0/1344320201 reap_dead start 2023-09-30T00:18:02.096 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fe3e0375700 1 -- 172.21.15.94:0/2435658022 shutdown_connections 2023-09-30T00:18:02.096 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fe3e0375700 1 -- 172.21.15.94:0/2435658022 >> 172.21.15.94:0/2435658022 conn(0x7fe3d80b3520 msgr2=0x7fe3d809cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7f872063c700 1 -- 172.21.15.94:0/1344320201 shutdown_connections 2023-09-30T00:18:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7f872063c700 1 -- 172.21.15.94:0/1344320201 >> 172.21.15.94:0/1344320201 conn(0x7f87180b3520 msgr2=0x7f871809d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7f872063c700 1 -- 172.21.15.94:0/1344320201 shutdown_connections 2023-09-30T00:18:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7f872063c700 1 -- 172.21.15.94:0/1344320201 wait complete. 2023-09-30T00:18:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.063+0000 7fccf37fe700 1 -- 172.21.15.94:0/2924068270 <== osd.0 v2:172.21.15.94:6800/84364 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcd0c13d5a0 con 0x7fccf80a02e0 2023-09-30T00:18:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fe3e0375700 1 -- 172.21.15.94:0/2435658022 shutdown_connections 2023-09-30T00:18:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7f5f3911d700 1 -- 172.21.15.94:0/802123000 --> [v2:172.21.15.121:6800/86820,v1:172.21.15.121:6801/86820] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5f3414c9a0 con 0x7f5f200a03b0 2023-09-30T00:18:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fe3e0375700 1 -- 172.21.15.94:0/2435658022 wait complete. 2023-09-30T00:18:02.099 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:6800/84364,v1:172.21.15.94:6801/84364] conn(0x7fccf80a02e0 msgr2=0x7fccf80a2710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.099 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fcd10bd3700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:6800/84364,v1:172.21.15.94:6801/84364] conn(0x7fccf80a02e0 0x7fccf80a2710 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.099 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fcd10bd3700 10 mgrc shutdown 2023-09-30T00:18:02.099 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fccf8098780 msgr2=0x7fccf809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.099 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fcd10bd3700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fccf8098780 0x7fccf809ac30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fccfc007fd0 tx=0x7fccfc007f60 comp rx=0 tx=0).stop 2023-09-30T00:18:02.099 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2924068270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0c140bc0 msgr2=0x7fcd0c13a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.099 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fcd10bd3700 1 --2- 172.21.15.94:0/2924068270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0c140bc0 0x7fcd0c13a480 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcd04007ed0 tx=0x7fcd04009850 comp rx=0 tx=0).stop 2023-09-30T00:18:02.100 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7f5f1bfff700 1 -- 172.21.15.94:0/802123000 <== osd.2 v2:172.21.15.121:6800/86820 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5f3414c9a0 con 0x7f5f200a03b0 2023-09-30T00:18:02.100 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fcd0bfff700 1 -- 172.21.15.94:0/2924068270 reap_dead start 2023-09-30T00:18:02.100 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2924068270 shutdown_connections 2023-09-30T00:18:02.100 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2924068270 >> 172.21.15.94:0/2924068270 conn(0x7fcd0c0b43f0 msgr2=0x7fcd0c0a3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.101 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2924068270 shutdown_connections 2023-09-30T00:18:02.101 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7fcd10bd3700 1 -- 172.21.15.94:0/2924068270 wait complete. 2023-09-30T00:18:02.102 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7f5f3911d700 1 -- 172.21.15.94:0/802123000 >> [v2:172.21.15.121:6800/86820,v1:172.21.15.121:6801/86820] conn(0x7f5f200a03b0 msgr2=0x7f5f200a27e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.102 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7f5f3911d700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.121:6800/86820,v1:172.21.15.121:6801/86820] conn(0x7f5f200a03b0 0x7f5f200a27e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.102 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7f5f3911d700 10 mgrc shutdown 2023-09-30T00:18:02.102 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7f5f3911d700 1 -- 172.21.15.94:0/802123000 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5f20098850 msgr2=0x7f5f2009ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.102 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7f5f3911d700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5f20098850 0x7f5f2009ad00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400c240 tx=0x7f5f24015000 comp rx=0 tx=0).stop 2023-09-30T00:18:02.102 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7f5f3911d700 1 -- 172.21.15.94:0/802123000 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f34139870 msgr2=0x7f5f3406b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.102 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.064+0000 7f5f3911d700 1 --2- 172.21.15.94:0/802123000 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5f34139870 0x7f5f3406b400 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5f340a27c0 tx=0x7f5f2c00b040 comp rx=0 tx=0).stop 2023-09-30T00:18:02.103 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.065+0000 7f5f3891c700 1 -- 172.21.15.94:0/802123000 reap_dead start 2023-09-30T00:18:02.103 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.065+0000 7f5f3911d700 1 -- 172.21.15.94:0/802123000 shutdown_connections 2023-09-30T00:18:02.103 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.065+0000 7f5f3911d700 1 -- 172.21.15.94:0/802123000 >> 172.21.15.94:0/802123000 conn(0x7f5f34070dc0 msgr2=0x7f5f340b5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.103 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.065+0000 7f5f3911d700 1 -- 172.21.15.94:0/802123000 shutdown_connections 2023-09-30T00:18:02.103 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.065+0000 7f5f3911d700 1 -- 172.21.15.94:0/802123000 wait complete. 2023-09-30T00:18:02.104 INFO:teuthology.orchestra.run.smithi094.stdout:42949672967 2023-09-30T00:18:02.104 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:18:02.105 INFO:teuthology.orchestra.run.smithi094.stdout:42949672967 2023-09-30T00:18:02.105 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:18:02.106 INFO:teuthology.orchestra.run.smithi094.stdout:38654705670 2023-09-30T00:18:02.106 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:18:02.107 INFO:teuthology.orchestra.run.smithi094.stdout:42949672967 2023-09-30T00:18:02.107 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:18:02.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f1c0fb31700 1 Processor -- start 2023-09-30T00:18:02.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f2c64c7a700 1 Processor -- start 2023-09-30T00:18:02.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f2c64c7a700 1 -- start start 2023-09-30T00:18:02.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f1c0fb31700 1 -- start start 2023-09-30T00:18:02.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f4a8323b700 1 Processor -- start 2023-09-30T00:18:02.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f4a8323b700 1 -- start start 2023-09-30T00:18:02.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f526d7ec700 1 Processor -- start 2023-09-30T00:18:02.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f526d7ec700 1 -- start start 2023-09-30T00:18:02.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f1c0fb31700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1c0809edb0 0x7f1c0809db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f1c0fb31700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 0x7f1c0806e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f1c0fb31700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c081419c0 con 0x7f1c08068630 2023-09-30T00:18:02.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f1c0fb31700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0806f2c0 con 0x7f1c0809edb0 2023-09-30T00:18:02.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.267+0000 7f1c0fb31700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c08140aa0 con 0x7f1c0809f180 2023-09-30T00:18:02.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f1c0d0cc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 0x7f1c0806e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f1c0d0cc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 0x7f1c0806e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39198/0 (socket says 172.21.15.94:39198) 2023-09-30T00:18:02.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f1c0d0cc700 1 -- 172.21.15.94:0/626963214 learned_addr learned my addr 172.21.15.94:0/626963214 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f2c64c7a700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2c600a2020 0x7f2c600a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f2c64c7a700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2c600a0e40 0x7f2c600b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f2c64c7a700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c600b7270 con 0x7f2c600b5e60 2023-09-30T00:18:02.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f2c64c7a700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c600a18d0 con 0x7f2c600a2020 2023-09-30T00:18:02.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f2c64c7a700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c60140540 con 0x7f2c600a0e40 2023-09-30T00:18:02.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f1c0d8cd700 1 --2- 172.21.15.94:0/626963214 >> v2:172.21.15.121:3300/0 conn(0x7f1c0809edb0 0x7f1c0809db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f2c5effd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2c600b5e60 0x7f2c600a0730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50250/0 (socket says 172.21.15.94:50250) 2023-09-30T00:18:02.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f2c5effd700 1 -- 172.21.15.94:0/654551738 learned_addr learned my addr 172.21.15.94:0/654551738 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:02.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f2c5f7fe700 1 --2- 172.21.15.94:0/654551738 >> v2:172.21.15.94:3300/0 conn(0x7f2c600a2020 0x7f2c600a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f2c5ffff700 1 --2- 172.21.15.94:0/654551738 >> v2:172.21.15.121:3300/0 conn(0x7f2c600a0e40 0x7f2c600b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a8323b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4a7c09bb30 0x7f4a7c13ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a8323b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4a7c09b340 0x7f4a7c143b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a8323b700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7c070910 con 0x7f4a7c070540 2023-09-30T00:18:02.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a8323b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a7c1404e0 con 0x7f4a7c09bb30 2023-09-30T00:18:02.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a8323b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c144090 con 0x7f4a7c09b340 2023-09-30T00:18:02.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a82a3a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4a7c09b340 0x7f4a7c143b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a82a3a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4a7c09b340 0x7f4a7c143b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39206/0 (socket says 172.21.15.94:39206) 2023-09-30T00:18:02.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a82a3a700 1 -- 172.21.15.94:0/2878646304 learned_addr learned my addr 172.21.15.94:0/2878646304 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f1c0d0cc700 1 -- 172.21.15.94:0/626963214 >> v2:172.21.15.121:3300/0 conn(0x7f1c0809edb0 msgr2=0x7f1c0809db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:02.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f1c0d0cc700 1 --2- 172.21.15.94:0/626963214 >> v2:172.21.15.121:3300/0 conn(0x7f1c0809edb0 0x7f1c0809db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f1c0d0cc700 1 -- 172.21.15.94:0/626963214 >> v1:172.21.15.94:6789/0 conn(0x7f1c08068630 legacy=0x7f1c081412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f1c0d0cc700 1 -- 172.21.15.94:0/626963214 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c08140d60 con 0x7f1c0809f180 2023-09-30T00:18:02.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f1c0d0cc700 1 --2- 172.21.15.94:0/626963214 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 0x7f1c0806e990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800bc50 tx=0x7f1bf800b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cda292cca2207808 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f526d7ec700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f52680702f0 0x7f526809e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f526d7ec700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52680706c0 0x7f5268067620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f526d7ec700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f526809e8b0 0x7f5268067b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a82a3a700 1 -- 172.21.15.94:0/2878646304 >> v2:172.21.15.94:3300/0 conn(0x7f4a7c09bb30 msgr2=0x7f4a7c13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:02.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f526d7ec700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f526809d9b0 con 0x7f526809e8b0 2023-09-30T00:18:02.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a82a3a700 1 --2- 172.21.15.94:0/2878646304 >> v2:172.21.15.94:3300/0 conn(0x7f4a7c09bb30 0x7f4a7c13ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a82a3a700 1 -- 172.21.15.94:0/2878646304 >> v1:172.21.15.94:6789/0 conn(0x7f4a7c070540 legacy=0x7f4a7c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f526d7ec700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52680681a0 con 0x7f52680702f0 2023-09-30T00:18:02.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f526d7ec700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5268068460 con 0x7f52680706c0 2023-09-30T00:18:02.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a82a3a700 1 -- 172.21.15.94:0/2878646304 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c099fd0 con 0x7f4a7c09b340 2023-09-30T00:18:02.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f5266ffd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f52680702f0 0x7f526809e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f5266ffd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f52680702f0 0x7f526809e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.94:44660/0 (socket says 172.21.15.94:44660) 2023-09-30T00:18:02.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f5266ffd700 1 -- 172.21.15.94:0/2639531044 learned_addr learned my addr 172.21.15.94:0/2639531044 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f52667fc700 1 --2- 172.21.15.94:0/2639531044 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52680706c0 0x7f5268067620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f52677fe700 1 --2- 172.21.15.94:0/2639531044 >> v2:172.21.15.94:3300/0 conn(0x7f526809e8b0 0x7f5268067b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f1c0c8cb700 1 -- 172.21.15.94:0/626963214 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf800be60 con 0x7f1c0809f180 2023-09-30T00:18:02.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f1c0c8cb700 1 -- 172.21.15.94:0/626963214 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf800de00 con 0x7f1c0809f180 2023-09-30T00:18:02.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f1c0c8cb700 1 -- 172.21.15.94:0/626963214 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8019690 con 0x7f1c0809f180 2023-09-30T00:18:02.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f2c5f7fe700 1 -- 172.21.15.94:0/654551738 >> v2:172.21.15.121:3300/0 conn(0x7f2c600a0e40 msgr2=0x7f2c600b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:02.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f2c5f7fe700 1 --2- 172.21.15.94:0/654551738 >> v2:172.21.15.121:3300/0 conn(0x7f2c600a0e40 0x7f2c600b62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f2c5f7fe700 1 -- 172.21.15.94:0/654551738 >> v1:172.21.15.94:6789/0 conn(0x7f2c600b5e60 legacy=0x7f2c600a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.268+0000 7f4a82a3a700 1 --2- 172.21.15.94:0/2878646304 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4a7c09b340 0x7f4a7c143b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009f20 tx=0x7f4a74009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79b8d4364407005b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f2c5f7fe700 1 -- 172.21.15.94:0/654551738 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c601407a0 con 0x7f2c600a2020 2023-09-30T00:18:02.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f2c5ffff700 1 --2- 172.21.15.94:0/654551738 >> v2:172.21.15.121:3300/0 conn(0x7f2c600a0e40 0x7f2c600b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:02.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f5266ffd700 1 -- 172.21.15.94:0/2639531044 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52680706c0 msgr2=0x7f5268067620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:02.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f5266ffd700 1 --2- 172.21.15.94:0/2639531044 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52680706c0 0x7f5268067620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.280 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f5266ffd700 1 -- 172.21.15.94:0/2639531044 >> v2:172.21.15.94:3300/0 conn(0x7f526809e8b0 msgr2=0x7f5268067b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:02.280 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f5266ffd700 1 --2- 172.21.15.94:0/2639531044 >> v2:172.21.15.94:3300/0 conn(0x7f526809e8b0 0x7f5268067b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.280 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f5266ffd700 1 -- 172.21.15.94:0/2639531044 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526809fa10 con 0x7f52680702f0 2023-09-30T00:18:02.280 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f2c5f7fe700 1 --2- 172.21.15.94:0/654551738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c600a2020 0x7f2c600a23f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2c600b74f0 tx=0x7f2c500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba43a64475c159f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.280 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f4a81237700 1 -- 172.21.15.94:0/2878646304 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7400e070 con 0x7f4a7c09b340 2023-09-30T00:18:02.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f4a81237700 1 -- 172.21.15.94:0/2878646304 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7400ba80 con 0x7f4a7c09b340 2023-09-30T00:18:02.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f4a81237700 1 -- 172.21.15.94:0/2878646304 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a740092e0 con 0x7f4a7c09b340 2023-09-30T00:18:02.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f5266ffd700 1 --2- 172.21.15.94:0/2639531044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52680702f0 0x7f526809e370 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f526806b650 tx=0x7f5258009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2532b9d9b5ef822 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f2c5e7fc700 1 -- 172.21.15.94:0/654551738 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50011630 con 0x7f2c600a2020 2023-09-30T00:18:02.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f2c5e7fc700 1 -- 172.21.15.94:0/654551738 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c50011790 con 0x7f2c600a2020 2023-09-30T00:18:02.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f2c5e7fc700 1 -- 172.21.15.94:0/654551738 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50011b30 con 0x7f2c600a2020 2023-09-30T00:18:02.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f5265ffb700 1 -- 172.21.15.94:0/2639531044 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258009e70 con 0x7f52680702f0 2023-09-30T00:18:02.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f5265ffb700 1 -- 172.21.15.94:0/2639531044 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525800b860 con 0x7f52680702f0 2023-09-30T00:18:02.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f5265ffb700 1 -- 172.21.15.94:0/2639531044 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525800bc00 con 0x7f52680702f0 2023-09-30T00:18:02.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f1c0fb31700 1 -- 172.21.15.94:0/626963214 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 msgr2=0x7f1c0806e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f1c0fb31700 1 --2- 172.21.15.94:0/626963214 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 0x7f1c0806e990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800bc50 tx=0x7f1bf800b4f0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f1c0fb31700 1 -- 172.21.15.94:0/626963214 shutdown_connections 2023-09-30T00:18:02.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f1c0fb31700 1 --2- 172.21.15.94:0/626963214 >> v2:172.21.15.121:3300/0 conn(0x7f1c0809edb0 0x7f1c0809db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f1c0fb31700 1 --2- 172.21.15.94:0/626963214 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 0x7f1c0806e990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f1c0fb31700 1 -- 172.21.15.94:0/626963214 >> 172.21.15.94:0/626963214 conn(0x7f1c080b29c0 msgr2=0x7f1c080b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f1c0fb31700 1 -- 172.21.15.94:0/626963214 shutdown_connections 2023-09-30T00:18:02.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.269+0000 7f1c0fb31700 1 -- 172.21.15.94:0/626963214 wait complete. 2023-09-30T00:18:02.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f4a8323b700 1 -- 172.21.15.94:0/2878646304 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4a7c09b340 msgr2=0x7f4a7c143b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f2c64c7a700 1 -- 172.21.15.94:0/654551738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c600a2020 msgr2=0x7f2c600a23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f4a8323b700 1 --2- 172.21.15.94:0/2878646304 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4a7c09b340 0x7f4a7c143b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009f20 tx=0x7f4a74009840 comp rx=0 tx=0).stop 2023-09-30T00:18:02.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f2c64c7a700 1 --2- 172.21.15.94:0/654551738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c600a2020 0x7f2c600a23f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2c600b74f0 tx=0x7f2c500091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f4a8323b700 1 -- 172.21.15.94:0/2878646304 shutdown_connections 2023-09-30T00:18:02.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f4a8323b700 1 --2- 172.21.15.94:0/2878646304 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4a7c09b340 0x7f4a7c143b50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f4a8323b700 1 --2- 172.21.15.94:0/2878646304 >> v2:172.21.15.94:3300/0 conn(0x7f4a7c09bb30 0x7f4a7c13ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f4a8323b700 1 -- 172.21.15.94:0/2878646304 >> 172.21.15.94:0/2878646304 conn(0x7f4a7c0a3570 msgr2=0x7f4a7c0a3970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f2c64c7a700 1 -- 172.21.15.94:0/654551738 shutdown_connections 2023-09-30T00:18:02.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f2c64c7a700 1 --2- 172.21.15.94:0/654551738 >> v2:172.21.15.121:3300/0 conn(0x7f2c600a0e40 0x7f2c600b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f2c64c7a700 1 --2- 172.21.15.94:0/654551738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c600a2020 0x7f2c600a23f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f4a8323b700 1 -- 172.21.15.94:0/2878646304 shutdown_connections 2023-09-30T00:18:02.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f4a8323b700 1 -- 172.21.15.94:0/2878646304 wait complete. 2023-09-30T00:18:02.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f2c64c7a700 1 -- 172.21.15.94:0/654551738 >> 172.21.15.94:0/654551738 conn(0x7f2c60070dc0 msgr2=0x7f2c60067bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f2c64c7a700 1 -- 172.21.15.94:0/654551738 shutdown_connections 2023-09-30T00:18:02.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f2c64c7a700 1 -- 172.21.15.94:0/654551738 wait complete. 2023-09-30T00:18:02.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f526d7ec700 1 -- 172.21.15.94:0/2639531044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52680702f0 msgr2=0x7f526809e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f526d7ec700 1 --2- 172.21.15.94:0/2639531044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52680702f0 0x7f526809e370 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f526806b650 tx=0x7f5258009250 comp rx=0 tx=0).stop 2023-09-30T00:18:02.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f526d7ec700 1 -- 172.21.15.94:0/2639531044 shutdown_connections 2023-09-30T00:18:02.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f526d7ec700 1 --2- 172.21.15.94:0/2639531044 >> v2:172.21.15.94:3300/0 conn(0x7f526809e8b0 0x7f5268067b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f526d7ec700 1 --2- 172.21.15.94:0/2639531044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52680702f0 0x7f526809e370 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f526d7ec700 1 --2- 172.21.15.94:0/2639531044 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52680706c0 0x7f5268067620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f526d7ec700 1 -- 172.21.15.94:0/2639531044 >> 172.21.15.94:0/2639531044 conn(0x7f52680a35a0 msgr2=0x7f52680a39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f526d7ec700 1 -- 172.21.15.94:0/2639531044 shutdown_connections 2023-09-30T00:18:02.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.270+0000 7f526d7ec700 1 -- 172.21.15.94:0/2639531044 wait complete. 2023-09-30T00:18:02.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.271+0000 7f1c0fb31700 1 Processor -- start 2023-09-30T00:18:02.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.271+0000 7f1c0fb31700 1 -- start start 2023-09-30T00:18:02.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.271+0000 7f4a8323b700 1 Processor -- start 2023-09-30T00:18:02.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.271+0000 7f4a8323b700 1 -- start start 2023-09-30T00:18:02.290 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f2c64c7a700 1 Processor -- start 2023-09-30T00:18:02.290 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f2c64c7a700 1 -- start start 2023-09-30T00:18:02.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0fb31700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c08068630 0x7f1c08139b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0fb31700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0809edb0 0x7f1c0806cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0fb31700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 0x7f1c081386e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0fb31700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0806db40 con 0x7f1c08068630 2023-09-30T00:18:02.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0fb31700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c08139060 con 0x7f1c0809edb0 2023-09-30T00:18:02.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0fb31700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c08139320 con 0x7f1c0809f180 2023-09-30T00:18:02.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f526d7ec700 1 Processor -- start 2023-09-30T00:18:02.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0d8cd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c08068630 0x7f1c08139b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0d8cd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c08068630 0x7f1c08139b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50744/0 (socket says 172.21.15.94:50744) 2023-09-30T00:18:02.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0d8cd700 1 -- 172.21.15.94:0/2259340924 learned_addr learned my addr 172.21.15.94:0/2259340924 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0e0ce700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 0x7f1c081386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0d0cc700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0809edb0 0x7f1c0806cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0d8cd700 1 -- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 msgr2=0x7f1c081386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0d8cd700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 0x7f1c081386e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0d8cd700 1 -- 172.21.15.94:0/2259340924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0809edb0 msgr2=0x7f1c0806cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0d8cd700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0809edb0 0x7f1c0806cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0d8cd700 1 -- 172.21.15.94:0/2259340924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf800b790 con 0x7f1c08068630 2023-09-30T00:18:02.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0e0ce700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 0x7f1c081386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:02.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f4a8323b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a7c070540 0x7f4a7c09ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f1c0d8cd700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c08068630 0x7f1c08139b30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400d890 tx=0x7f1c04008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.294 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f2c64c7a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c600a0e40 0x7f2c600997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.294 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f4a8323b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a7c09b340 0x7f4a7c09d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.294 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f4a8323b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4a7c09bb30 0x7f4a7c09d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.294 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f2c64c7a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c600a2020 0x7f2c6009ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.295 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f4a8323b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c09ef60 con 0x7f4a7c070540 2023-09-30T00:18:02.295 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f4a8323b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c09dfd0 con 0x7f4a7c09b340 2023-09-30T00:18:02.295 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f4a8323b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c09da90 con 0x7f4a7c09bb30 2023-09-30T00:18:02.295 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f2c64c7a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2c600b5e60 0x7f2c60096e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f2c64c7a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6009b420 con 0x7f2c600a2020 2023-09-30T00:18:02.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f2c64c7a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c601514f0 con 0x7f2c600a0e40 2023-09-30T00:18:02.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f2c64c7a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c601517b0 con 0x7f2c600b5e60 2023-09-30T00:18:02.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f4a82239700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a7c070540 0x7f4a7c09ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f4a81a38700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a7c09b340 0x7f4a7c09d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f4a82239700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a7c070540 0x7f4a7c09ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50752/0 (socket says 172.21.15.94:50752) 2023-09-30T00:18:02.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f4a81a38700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a7c09b340 0x7f4a7c09d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.94:44684/0 (socket says 172.21.15.94:44684) 2023-09-30T00:18:02.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f4a82239700 1 -- 172.21.15.94:0/3667479079 learned_addr learned my addr 172.21.15.94:0/3667479079 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f2c5effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c600a2020 0x7f2c6009ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f2c5effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c600a2020 0x7f2c6009ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50754/0 (socket says 172.21.15.94:50754) 2023-09-30T00:18:02.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f2c5effd700 1 -- 172.21.15.94:0/133014690 learned_addr learned my addr 172.21.15.94:0/133014690 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f2c5f7fe700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c600a0e40 0x7f2c600997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f2c5ffff700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2c600b5e60 0x7f2c60096e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.272+0000 7f526d7ec700 1 -- start start 2023-09-30T00:18:02.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f526d7ec700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52680702f0 0x7f5268097c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f526d7ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52680706c0 0x7f5268140e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f526d7ec700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f526809e8b0 0x7f5268141350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f526d7ec700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5268141890 con 0x7f526809e8b0 2023-09-30T00:18:02.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f526d7ec700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52680981d0 con 0x7f52680706c0 2023-09-30T00:18:02.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f526d7ec700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f526806e290 con 0x7f52680702f0 2023-09-30T00:18:02.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f5266ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52680702f0 0x7f5268097c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f5266ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52680702f0 0x7f5268097c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39258/0 (socket says 172.21.15.94:39258) 2023-09-30T00:18:02.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f5266ffd700 1 -- 172.21.15.94:0/3079924886 learned_addr learned my addr 172.21.15.94:0/3079924886 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:02.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f2c5f7fe700 1 -- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2c600b5e60 msgr2=0x7f2c60096e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f2c5f7fe700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2c600b5e60 0x7f2c60096e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f2c5f7fe700 1 -- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c600a2020 msgr2=0x7f2c6009ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f2c5f7fe700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c600a2020 0x7f2c6009ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f2c5f7fe700 1 -- 172.21.15.94:0/133014690 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c58009a20 con 0x7f2c600a0e40 2023-09-30T00:18:02.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f4a82a3a700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4a7c09bb30 0x7f4a7c09d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f4a82239700 1 -- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4a7c09bb30 msgr2=0x7f4a7c09d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f4a82239700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4a7c09bb30 0x7f4a7c09d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f4a82239700 1 -- 172.21.15.94:0/3667479079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a7c09b340 msgr2=0x7f4a7c09d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f4a82239700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a7c09b340 0x7f4a7c09d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f4a82239700 1 -- 172.21.15.94:0/3667479079 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a74009ae0 con 0x7f4a7c070540 2023-09-30T00:18:02.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f52667fc700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52680706c0 0x7f5268140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f52677fe700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f526809e8b0 0x7f5268141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f4a82239700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a7c070540 0x7f4a7c09ea20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800c590 tx=0x7f4a7800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f5266ffd700 1 -- 172.21.15.94:0/3079924886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52680706c0 msgr2=0x7f5268140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f5266ffd700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52680706c0 0x7f5268140e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f5266ffd700 1 -- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f526809e8b0 msgr2=0x7f5268141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f5266ffd700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f526809e8b0 0x7f5268141350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f2c5f7fe700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c600a0e40 0x7f2c600997d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c600a2930 tx=0x7f2c50017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f5266ffd700 1 -- 172.21.15.94:0/3079924886 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52580094f0 con 0x7f52680702f0 2023-09-30T00:18:02.302 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f2c477fe700 1 -- 172.21.15.94:0/133014690 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50011b10 con 0x7f2c600a0e40 2023-09-30T00:18:02.302 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f5266ffd700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52680702f0 0x7f5268097c90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f52680682e0 tx=0x7f5258017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f2c477fe700 1 -- 172.21.15.94:0/133014690 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c500253f0 con 0x7f2c600a0e40 2023-09-30T00:18:02.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f2c477fe700 1 -- 172.21.15.94:0/133014690 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50025790 con 0x7f2c600a0e40 2023-09-30T00:18:02.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f2c64c7a700 1 -- 172.21.15.94:0/133014690 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c50009420 con 0x7f2c600a0e40 2023-09-30T00:18:02.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f2c64c7a700 1 -- 172.21.15.94:0/133014690 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6013e670 con 0x7f2c600a0e40 2023-09-30T00:18:02.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f524effd700 1 -- 172.21.15.94:0/3079924886 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525800bbe0 con 0x7f52680702f0 2023-09-30T00:18:02.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f524effd700 1 -- 172.21.15.94:0/3079924886 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5258000c90 con 0x7f52680702f0 2023-09-30T00:18:02.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f526d7ec700 1 -- 172.21.15.94:0/3079924886 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52681549b0 con 0x7f52680702f0 2023-09-30T00:18:02.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f524effd700 1 -- 172.21.15.94:0/3079924886 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258011650 con 0x7f52680702f0 2023-09-30T00:18:02.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f526d7ec700 1 -- 172.21.15.94:0/3079924886 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5268154fb0 con 0x7f52680702f0 2023-09-30T00:18:02.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f1bfdffb700 1 -- 172.21.15.94:0/2259340924 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c040125f0 con 0x7f1c08068630 2023-09-30T00:18:02.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f1bfdffb700 1 -- 172.21.15.94:0/2259340924 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c04012750 con 0x7f1c08068630 2023-09-30T00:18:02.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.273+0000 7f1bfdffb700 1 -- 172.21.15.94:0/2259340924 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04012ad0 con 0x7f1c08068630 2023-09-30T00:18:02.305 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f1c0fb31700 1 -- 172.21.15.94:0/2259340924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0806d4b0 con 0x7f1c08068630 2023-09-30T00:18:02.305 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f1c0fb31700 1 -- 172.21.15.94:0/2259340924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c08138c20 con 0x7f1c08068630 2023-09-30T00:18:02.305 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.274+0000 7f4a727fc700 1 -- 172.21.15.94:0/3667479079 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a78011590 con 0x7f4a7c070540 2023-09-30T00:18:02.305 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.275+0000 7f4a727fc700 1 -- 172.21.15.94:0/3667479079 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a780116f0 con 0x7f4a7c070540 2023-09-30T00:18:02.305 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.275+0000 7f4a727fc700 1 -- 172.21.15.94:0/3667479079 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a78011ab0 con 0x7f4a7c070540 2023-09-30T00:18:02.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.275+0000 7f2c64c7a700 1 -- 172.21.15.94:0/133014690 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6004c380 con 0x7f2c600a0e40 2023-09-30T00:18:02.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.275+0000 7f526d7ec700 1 -- 172.21.15.94:0/3079924886 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f526804c380 con 0x7f52680702f0 2023-09-30T00:18:02.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.275+0000 7f4a8323b700 1 -- 172.21.15.94:0/3667479079 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7c06da00 con 0x7f4a7c070540 2023-09-30T00:18:02.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.275+0000 7f4a8323b700 1 -- 172.21.15.94:0/3667479079 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7c09f820 con 0x7f4a7c070540 2023-09-30T00:18:02.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.275+0000 7f1c0fb31700 1 -- 172.21.15.94:0/2259340924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c08138e10 con 0x7f1c08068630 2023-09-30T00:18:02.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.276+0000 7f4a8323b700 1 -- 172.21.15.94:0/3667479079 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7c09f150 con 0x7f4a7c070540 2023-09-30T00:18:02.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.276+0000 7f2c477fe700 1 -- 172.21.15.94:0/133014690 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2c5002e450 con 0x7f2c600a0e40 2023-09-30T00:18:02.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.276+0000 7f2c477fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:02.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.277+0000 7f2c477fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:02.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.277+0000 7f2c477fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.277+0000 7f2c477fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.277+0000 7f2c477fe700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2c4c098780 0x7f2c4c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.277+0000 7f1bfdffb700 1 -- 172.21.15.94:0/2259340924 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f1c0401e460 con 0x7f1c08068630 2023-09-30T00:18:02.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.277+0000 7f1bfdffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:02.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.277+0000 7f1bfdffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:02.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.277+0000 7f1bfdffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.277+0000 7f1bfdffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.277+0000 7f1bfdffb700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1bec098850 0x7f1bec09ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.278+0000 7f1bfdffb700 1 -- 172.21.15.94:0/2259340924 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f1c040c0a30 con 0x7f1c08068630 2023-09-30T00:18:02.309 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.278+0000 7f1c0d0cc700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1bec098850 0x7f1bec09ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.309 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.277+0000 7f2c477fe700 1 -- 172.21.15.94:0/133014690 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f2c500c1ba0 con 0x7f2c600a0e40 2023-09-30T00:18:02.309 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.277+0000 7f2c5effd700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2c4c098780 0x7f2c4c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.309 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.278+0000 7f2c5effd700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2c4c098780 0x7f2c4c09ac30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2c600b58d0 tx=0x7f2c54009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.309 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.278+0000 7f1c0d0cc700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1bec098850 0x7f1bec09ad00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800bc50 tx=0x7f1bf801b000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.278+0000 7f524effd700 1 -- 172.21.15.94:0/3079924886 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5258011a80 con 0x7f52680702f0 2023-09-30T00:18:02.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.278+0000 7f524effd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:02.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.278+0000 7f4a727fc700 1 -- 172.21.15.94:0/3667479079 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f4a78012420 con 0x7f4a7c070540 2023-09-30T00:18:02.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.278+0000 7f4a727fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:02.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f4a727fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:02.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f4a727fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f4a727fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f4a727fc700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4a60098700 0x7f4a6009abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f524effd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:02.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f524effd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f524effd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:02.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f524effd700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5254098700 0x7f525409abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:02.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f4a81a38700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4a60098700 0x7f4a6009abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f4a727fc700 1 -- 172.21.15.94:0/3667479079 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f4a780bfee0 con 0x7f4a7c070540 2023-09-30T00:18:02.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f52667fc700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5254098700 0x7f525409abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:02.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f524effd700 1 -- 172.21.15.94:0/3079924886 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f52580c1b00 con 0x7f52680702f0 2023-09-30T00:18:02.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f4a81a38700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4a60098700 0x7f4a6009abb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c069bb0 tx=0x7f4a6c0090f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.279+0000 7f52667fc700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5254098700 0x7f525409abb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f52680b3d50 tx=0x7f525c0091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:02.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.281+0000 7f524effd700 1 -- 172.21.15.94:0/3079924886 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f525808f240 con 0x7f52680702f0 2023-09-30T00:18:02.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.282+0000 7f4a727fc700 1 -- 172.21.15.94:0/3667479079 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4a7808d620 con 0x7f4a7c070540 2023-09-30T00:18:02.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.282+0000 7f2c477fe700 1 -- 172.21.15.94:0/133014690 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c5008f250 con 0x7f2c600a0e40 2023-09-30T00:18:02.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.286+0000 7f1bfdffb700 1 -- 172.21.15.94:0/2259340924 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c0408e0e0 con 0x7f1c08068630 2023-09-30T00:18:02.476 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.475+0000 7f4a8323b700 1 -- 172.21.15.94:0/3667479079 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4a7c06dbf0 con 0x7f4a7c070540 2023-09-30T00:18:02.476 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.476+0000 7f4a727fc700 1 -- 172.21.15.94:0/3667479079 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4a78021020 con 0x7f4a7c070540 2023-09-30T00:18:02.476 INFO:teuthology.orchestra.run.smithi094.stdout:38654705668 2023-09-30T00:18:02.479 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 10 mgrc shutdown 2023-09-30T00:18:02.479 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 1 -- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4a60098700 msgr2=0x7f4a6009abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.479 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4a60098700 0x7f4a6009abb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c069bb0 tx=0x7f4a6c0090f0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.479 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 1 -- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a7c070540 msgr2=0x7f4a7c09ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.479 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a7c070540 0x7f4a7c09ea20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800c590 tx=0x7f4a7800c8a0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.479 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 1 -- 172.21.15.94:0/3667479079 shutdown_connections 2023-09-30T00:18:02.479 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4a60098700 0x7f4a6009abb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.479 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4a7c09bb30 0x7f4a7c09d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.479 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a7c070540 0x7f4a7c09ea20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.480 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 1 --2- 172.21.15.94:0/3667479079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a7c09b340 0x7f4a7c09d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.480 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 1 -- 172.21.15.94:0/3667479079 >> 172.21.15.94:0/3667479079 conn(0x7f4a7c0a3570 msgr2=0x7f4a7c13f120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.480 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f1c0fb31700 1 -- 172.21.15.94:0/2259340924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1c0813a070 con 0x7f1c08068630 2023-09-30T00:18:02.480 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 1 -- 172.21.15.94:0/3667479079 shutdown_connections 2023-09-30T00:18:02.480 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.478+0000 7f4a8323b700 1 -- 172.21.15.94:0/3667479079 wait complete. 2023-09-30T00:18:02.481 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.479+0000 7f1bfdffb700 1 -- 172.21.15.94:0/2259340924 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1c04091700 con 0x7f1c08068630 2023-09-30T00:18:02.481 INFO:teuthology.orchestra.run.smithi094.stdout:42949672966 2023-09-30T00:18:02.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 10 mgrc shutdown 2023-09-30T00:18:02.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 1 -- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1bec098850 msgr2=0x7f1bec09ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1bec098850 0x7f1bec09ad00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800bc50 tx=0x7f1bf801b000 comp rx=0 tx=0).stop 2023-09-30T00:18:02.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 1 -- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c08068630 msgr2=0x7f1c08139b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c08068630 0x7f1c08139b30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400d890 tx=0x7f1c04008040 comp rx=0 tx=0).stop 2023-09-30T00:18:02.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 1 -- 172.21.15.94:0/2259340924 shutdown_connections 2023-09-30T00:18:02.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1bec098850 0x7f1bec09ad00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1c0809f180 0x7f1c081386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c08068630 0x7f1c08139b30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 1 --2- 172.21.15.94:0/2259340924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c0809edb0 0x7f1c0806cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 1 -- 172.21.15.94:0/2259340924 >> 172.21.15.94:0/2259340924 conn(0x7f1c080b29c0 msgr2=0x7f1c080b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 1 -- 172.21.15.94:0/2259340924 shutdown_connections 2023-09-30T00:18:02.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.481+0000 7f1c0fb31700 1 -- 172.21.15.94:0/2259340924 wait complete. 2023-09-30T00:18:02.492 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705668 for osd.0 2023-09-30T00:18:02.492 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.492+0000 7f526d7ec700 1 -- 172.21.15.94:0/3079924886 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5268154fb0 con 0x7f52680702f0 2023-09-30T00:18:02.493 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.492+0000 7f524effd700 1 -- 172.21.15.94:0/3079924886 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5268154fb0 con 0x7f52680702f0 2023-09-30T00:18:02.493 INFO:teuthology.orchestra.run.smithi094.stdout:42949672966 2023-09-30T00:18:02.495 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.494+0000 7f526d7ec700 10 mgrc shutdown 2023-09-30T00:18:02.495 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.494+0000 7f526d7ec700 1 -- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5254098700 msgr2=0x7f525409abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.495 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.494+0000 7f526d7ec700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5254098700 0x7f525409abb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f52680b3d50 tx=0x7f525c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.495 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.494+0000 7f526d7ec700 1 -- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52680702f0 msgr2=0x7f5268097c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.495 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.494+0000 7f526d7ec700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52680702f0 0x7f5268097c90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f52680682e0 tx=0x7f5258017000 comp rx=0 tx=0).stop 2023-09-30T00:18:02.495 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.494+0000 7f526d7ec700 1 -- 172.21.15.94:0/3079924886 shutdown_connections 2023-09-30T00:18:02.495 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.494+0000 7f526d7ec700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5254098700 0x7f525409abb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.496 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.494+0000 7f526d7ec700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52680702f0 0x7f5268097c90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.496 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.494+0000 7f526d7ec700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f526809e8b0 0x7f5268141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.496 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.494+0000 7f526d7ec700 1 --2- 172.21.15.94:0/3079924886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52680706c0 0x7f5268140e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.496 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.494+0000 7f526d7ec700 1 -- 172.21.15.94:0/3079924886 >> 172.21.15.94:0/3079924886 conn(0x7f52680a35a0 msgr2=0x7f52680a0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.496 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.494+0000 7f526d7ec700 1 -- 172.21.15.94:0/3079924886 shutdown_connections 2023-09-30T00:18:02.496 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.495+0000 7f526d7ec700 1 -- 172.21.15.94:0/3079924886 wait complete. 2023-09-30T00:18:02.498 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.498+0000 7f2c64c7a700 1 -- 172.21.15.94:0/133014690 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2c60140a60 con 0x7f2c600a0e40 2023-09-30T00:18:02.499 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.499+0000 7f2c477fe700 1 -- 172.21.15.94:0/133014690 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2c50015070 con 0x7f2c600a0e40 2023-09-30T00:18:02.499 INFO:teuthology.orchestra.run.smithi094.stdout:42949672966 2023-09-30T00:18:02.500 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672966 for osd.1 2023-09-30T00:18:02.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 10 mgrc shutdown 2023-09-30T00:18:02.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 1 -- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2c4c098780 msgr2=0x7f2c4c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2c4c098780 0x7f2c4c09ac30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2c600b58d0 tx=0x7f2c54009040 comp rx=0 tx=0).stop 2023-09-30T00:18:02.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 1 -- 172.21.15.94:0/133014690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c600a0e40 msgr2=0x7f2c600997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:02.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c600a0e40 0x7f2c600997d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c600a2930 tx=0x7f2c50017000 comp rx=0 tx=0).stop 2023-09-30T00:18:02.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 1 -- 172.21.15.94:0/133014690 shutdown_connections 2023-09-30T00:18:02.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2c4c098780 0x7f2c4c09ac30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2c600b5e60 0x7f2c60096e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c600a0e40 0x7f2c600997d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 1 --2- 172.21.15.94:0/133014690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c600a2020 0x7f2c6009ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:02.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 1 -- 172.21.15.94:0/133014690 >> 172.21.15.94:0/133014690 conn(0x7f2c60070dc0 msgr2=0x7f2c60068a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:02.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 1 -- 172.21.15.94:0/133014690 shutdown_connections 2023-09-30T00:18:02.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:02.501+0000 7f2c64c7a700 1 -- 172.21.15.94:0/133014690 wait complete. 2023-09-30T00:18:02.508 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672966 for osd.2 2023-09-30T00:18:02.514 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672966 for osd.3 2023-09-30T00:18:03.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:18:03.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:18:03.509 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:18:03.514 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.617+0000 7fb4a0f06700 1 Processor -- start 2023-09-30T00:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.617+0000 7fb4a0f06700 1 -- start start 2023-09-30T00:18:03.620 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.618+0000 7fb4a0f06700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb49c0702f0 0x7fb49c0a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.620 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.618+0000 7fb4a0f06700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb49c0706c0 0x7fb49c13d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.620 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.618+0000 7fb4a0f06700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c0695f0 con 0x7fb49c0a0700 2023-09-30T00:18:03.620 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.618+0000 7fb4a0f06700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49c141c60 con 0x7fb49c0706c0 2023-09-30T00:18:03.620 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.618+0000 7fb4a0f06700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49c141f20 con 0x7fb49c0702f0 2023-09-30T00:18:03.620 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.618+0000 7fb49affd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb49c0706c0 0x7fb49c13d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.620 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.618+0000 7fb49affd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb49c0706c0 0x7fb49c13d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50768/0 (socket says 172.21.15.94:50768) 2023-09-30T00:18:03.620 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.618+0000 7fb49affd700 1 -- 172.21.15.94:0/3023731928 learned_addr learned my addr 172.21.15.94:0/3023731928 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:03.620 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.618+0000 7fb49b7fe700 1 --2- 172.21.15.94:0/3023731928 >> v2:172.21.15.121:3300/0 conn(0x7fb49c0702f0 0x7fb49c0a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.620 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.619+0000 7fb49affd700 1 -- 172.21.15.94:0/3023731928 >> v2:172.21.15.121:3300/0 conn(0x7fb49c0702f0 msgr2=0x7fb49c0a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:03.621 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.619+0000 7fb49affd700 1 --2- 172.21.15.94:0/3023731928 >> v2:172.21.15.121:3300/0 conn(0x7fb49c0702f0 0x7fb49c0a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.621 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.619+0000 7fb49affd700 1 -- 172.21.15.94:0/3023731928 >> v1:172.21.15.121:6789/0 conn(0x7fb49c0a0700 legacy=0x7fb49c067620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.621 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.619+0000 7fb49affd700 1 -- 172.21.15.94:0/3023731928 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c141940 con 0x7fb49c0706c0 2023-09-30T00:18:03.621 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.619+0000 7fb49affd700 1 --2- 172.21.15.94:0/3023731928 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb49c0706c0 0x7fb49c13d870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb49c063f50 tx=0x7fb490009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6501f7ec34711e12 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:03.621 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.619+0000 7fb49a7fc700 1 -- 172.21.15.94:0/3023731928 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49000b860 con 0x7fb49c0706c0 2023-09-30T00:18:03.621 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.619+0000 7fb49a7fc700 1 -- 172.21.15.94:0/3023731928 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49000b9c0 con 0x7fb49c0706c0 2023-09-30T00:18:03.621 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.619+0000 7fb49a7fc700 1 -- 172.21.15.94:0/3023731928 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49000bd60 con 0x7fb49c0706c0 2023-09-30T00:18:03.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.620+0000 7fb4a0f06700 1 -- 172.21.15.94:0/3023731928 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb49c0706c0 msgr2=0x7fb49c13d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.620+0000 7fb4a0f06700 1 --2- 172.21.15.94:0/3023731928 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb49c0706c0 0x7fb49c13d870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb49c063f50 tx=0x7fb490009250 comp rx=0 tx=0).stop 2023-09-30T00:18:03.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.620+0000 7fb4a0f06700 1 -- 172.21.15.94:0/3023731928 shutdown_connections 2023-09-30T00:18:03.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.620+0000 7fb4a0f06700 1 --2- 172.21.15.94:0/3023731928 >> v2:172.21.15.121:3300/0 conn(0x7fb49c0702f0 0x7fb49c0a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.620+0000 7fb4a0f06700 1 --2- 172.21.15.94:0/3023731928 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb49c0706c0 0x7fb49c13d870 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.620+0000 7fb4a0f06700 1 -- 172.21.15.94:0/3023731928 >> 172.21.15.94:0/3023731928 conn(0x7fb49c0a3570 msgr2=0x7fb49c0a3970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:03.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.620+0000 7fb4a0f06700 1 -- 172.21.15.94:0/3023731928 shutdown_connections 2023-09-30T00:18:03.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.620+0000 7fb4a0f06700 1 -- 172.21.15.94:0/3023731928 wait complete. 2023-09-30T00:18:03.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.621+0000 7fb4a0f06700 1 Processor -- start 2023-09-30T00:18:03.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.621+0000 7fb4a0f06700 1 -- start start 2023-09-30T00:18:03.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb4a0f06700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb49c0702f0 0x7fb49c06e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb4a0f06700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb49c0706c0 0x7fb49c09ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb4a0f06700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb49c0a0700 0x7fb49c09be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb4a0f06700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c09a710 con 0x7fb49c0a0700 2023-09-30T00:18:03.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb4a0f06700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c09c390 con 0x7fb49c0706c0 2023-09-30T00:18:03.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb4a0f06700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c09c880 con 0x7fb49c0702f0 2023-09-30T00:18:03.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb49b7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb49c0702f0 0x7fb49c06e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb49b7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb49c0702f0 0x7fb49c06e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39272/0 (socket says 172.21.15.94:39272) 2023-09-30T00:18:03.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb49b7fe700 1 -- 172.21.15.94:0/1348318651 learned_addr learned my addr 172.21.15.94:0/1348318651 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:03.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb49bfff700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb49c0a0700 0x7fb49c09be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb49affd700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb49c0706c0 0x7fb49c09ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb49bfff700 1 -- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb49c0702f0 msgr2=0x7fb49c06e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb49bfff700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb49c0702f0 0x7fb49c06e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb49bfff700 1 -- 172.21.15.94:0/1348318651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb49c0706c0 msgr2=0x7fb49c09ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb49bfff700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb49c0706c0 0x7fb49c09ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb49bfff700 1 -- 172.21.15.94:0/1348318651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4900094f0 con 0x7fb49c0a0700 2023-09-30T00:18:03.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb49b7fe700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb49c0702f0 0x7fb49c06e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:03.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.622+0000 7fb49bfff700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb49c0a0700 0x7fb49c09be50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb49c0a0b00 tx=0x7fb4940015f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:03.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.623+0000 7fb4837fe700 1 -- 172.21.15.94:0/1348318651 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494011590 con 0x7fb49c0a0700 2023-09-30T00:18:03.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.623+0000 7fb4837fe700 1 -- 172.21.15.94:0/1348318651 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb494011710 con 0x7fb49c0a0700 2023-09-30T00:18:03.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.623+0000 7fb4a0f06700 1 -- 172.21.15.94:0/1348318651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49c09c5c0 con 0x7fb49c0a0700 2023-09-30T00:18:03.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.623+0000 7fb4837fe700 1 -- 172.21.15.94:0/1348318651 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494011ad0 con 0x7fb49c0a0700 2023-09-30T00:18:03.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.623+0000 7fb4a0f06700 1 -- 172.21.15.94:0/1348318651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49c099e10 con 0x7fb49c0a0700 2023-09-30T00:18:03.628 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.624+0000 7fb4a0f06700 1 -- 172.21.15.94:0/1348318651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49c04c380 con 0x7fb49c0a0700 2023-09-30T00:18:03.628 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.626+0000 7fb4837fe700 1 -- 172.21.15.94:0/1348318651 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb494012420 con 0x7fb49c0a0700 2023-09-30T00:18:03.628 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.626+0000 7fb4837fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:03.628 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.626+0000 7fb4837fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:03.628 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.626+0000 7fb4837fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:03.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.626+0000 7fb4837fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:03.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.626+0000 7fb4837fe700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb488098700 0x7fb48809abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.626+0000 7fb49b7fe700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb488098700 0x7fb48809abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.626+0000 7fb4837fe700 1 -- 172.21.15.94:0/1348318651 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fb4940c0130 con 0x7fb49c0a0700 2023-09-30T00:18:03.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.627+0000 7fb49b7fe700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb488098700 0x7fb48809abb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb48c006fd0 tx=0x7fb48c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:03.631 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.631+0000 7fb4837fe700 1 -- 172.21.15.94:0/1348318651 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb49408d870 con 0x7fb49c0a0700 2023-09-30T00:18:03.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.659+0000 7fa2bef81700 1 Processor -- start 2023-09-30T00:18:03.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.659+0000 7fa2bef81700 1 -- start start 2023-09-30T00:18:03.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7eff115d3700 1 Processor -- start 2023-09-30T00:18:03.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7eff115d3700 1 -- start start 2023-09-30T00:18:03.661 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7fa2bef81700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8068a70 0x7fa2b80a26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.661 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7fa2bef81700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b8070910 con 0x7fa2b8070540 2023-09-30T00:18:03.661 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7fa2bef81700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b8067e20 con 0x7fa2b80691f0 2023-09-30T00:18:03.661 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7fa2bef81700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b80680e0 con 0x7fa2b8068a70 2023-09-30T00:18:03.661 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7fa2bd51e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8068a70 0x7fa2b80a26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.661 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7fa2bcd1d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa2b8070540 0x7fa2b80a0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50266/0 (socket says 172.21.15.94:50266) 2023-09-30T00:18:03.662 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7fa2bcd1d700 1 -- 172.21.15.94:0/2441659331 learned_addr learned my addr 172.21.15.94:0/2441659331 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:03.662 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7eff115d3700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7eff0c141450 0x7eff0c148c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.662 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7eff115d3700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c068e10 con 0x7eff0c068a40 2023-09-30T00:18:03.662 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7eff115d3700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c09d900 con 0x7eff0c140920 2023-09-30T00:18:03.662 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7eff115d3700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff0c09e4d0 con 0x7eff0c141450 2023-09-30T00:18:03.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.660+0000 7fa2bd51e700 1 -- 172.21.15.94:0/2441659331 >> v1:172.21.15.121:6789/0 conn(0x7fa2b80691f0 legacy=0x7fa2b8067620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:03.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7fa2bd51e700 1 -- 172.21.15.94:0/2441659331 >> v1:172.21.15.94:6789/0 conn(0x7fa2b8070540 legacy=0x7fa2b80a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7fa2bd51e700 1 -- 172.21.15.94:0/2441659331 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b80a2c30 con 0x7fa2b8068a70 2023-09-30T00:18:03.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff0affd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7eff0c068a40 0x7eff0c0a4de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50282/0 (socket says 172.21.15.94:50282) 2023-09-30T00:18:03.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff0affd700 1 -- 172.21.15.94:0/1426810040 learned_addr learned my addr 172.21.15.94:0/1426810040 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:03.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7fa2bd51e700 1 --2- 172.21.15.94:0/2441659331 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8068a70 0x7fa2b80a26f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009e90 tx=0x7fa2b0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3c4c130a5bf75b3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:03.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682112402 0 0) 0x7eff0c068e10 con 0x7eff0c068a40 2023-09-30T00:18:03.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef4003570 con 0x7eff0c068a40 2023-09-30T00:18:03.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803163232 0 0) 0x7eff0c09d900 con 0x7eff0c140920 2023-09-30T00:18:03.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff0c068e10 con 0x7eff0c140920 2023-09-30T00:18:03.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7fa2b77fe700 1 -- 172.21.15.94:0/2441659331 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b000e070 con 0x7fa2b8068a70 2023-09-30T00:18:03.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7fa2b77fe700 1 -- 172.21.15.94:0/2441659331 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b000bbf0 con 0x7fa2b8068a70 2023-09-30T00:18:03.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7fa2b77fe700 1 -- 172.21.15.94:0/2441659331 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0012430 con 0x7fa2b8068a70 2023-09-30T00:18:03.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1302675535 0 0) 0x7efef4003570 con 0x7eff0c068a40 2023-09-30T00:18:03.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 >> v2:172.21.15.121:3300/0 conn(0x7eff0c141450 msgr2=0x7eff0c148c00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-30T00:18:03.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 --2- 172.21.15.94:0/1426810040 >> v2:172.21.15.121:3300/0 conn(0x7eff0c141450 0x7eff0c148c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 >> v1:172.21.15.121:6789/0 conn(0x7eff0c140920 legacy=0x7eff0c140d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c09e790 con 0x7eff0c068a40 2023-09-30T00:18:03.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3153706020 0 0) 0x7efefc001470 con 0x7eff0c068a40 2023-09-30T00:18:03.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 >> v1:172.21.15.94:6789/0 conn(0x7eff0c068a40 legacy=0x7eff0c0a4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 --2- 172.21.15.94:0/1426810040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efef4004c80 0x7efef4005050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 --2- 172.21.15.94:0/1426810040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efef40037b0 0x7efef4003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 --2- 172.21.15.94:0/1426810040 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efef4004280 0x7efef400cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff00001230 con 0x7efef40037b0 2023-09-30T00:18:03.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c09e4d0 con 0x7efef4004c80 2023-09-30T00:18:03.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef400d390 con 0x7efef4004280 2023-09-30T00:18:03.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff0a7fc700 1 --2- 172.21.15.94:0/1426810040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efef40037b0 0x7efef4003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.661+0000 7eff0affd700 1 --2- 172.21.15.94:0/1426810040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efef4004c80 0x7efef4005050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7eff0a7fc700 1 -- 172.21.15.94:0/1426810040 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efef4004280 msgr2=0x7efef400cb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:18:03.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7eff0a7fc700 1 --2- 172.21.15.94:0/1426810040 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efef4004280 0x7efef400cb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7eff0a7fc700 1 -- 172.21.15.94:0/1426810040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efef4004c80 msgr2=0x7efef4005050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:03.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7eff0a7fc700 1 --2- 172.21.15.94:0/1426810040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efef4004c80 0x7efef4005050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7fa2bef81700 1 -- 172.21.15.94:0/2441659331 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8068a70 msgr2=0x7fa2b80a26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7eff0a7fc700 1 -- 172.21.15.94:0/1426810040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef4003570 con 0x7efef40037b0 2023-09-30T00:18:03.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7fa2bef81700 1 --2- 172.21.15.94:0/2441659331 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8068a70 0x7fa2b80a26f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009e90 tx=0x7fa2b0009780 comp rx=0 tx=0).stop 2023-09-30T00:18:03.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7fa2bef81700 1 -- 172.21.15.94:0/2441659331 shutdown_connections 2023-09-30T00:18:03.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7fa2bef81700 1 --2- 172.21.15.94:0/2441659331 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8068a70 0x7fa2b80a26f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.670 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7fa2bef81700 1 -- 172.21.15.94:0/2441659331 >> 172.21.15.94:0/2441659331 conn(0x7fa2b80a57c0 msgr2=0x7fa2b80a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:03.670 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7eff0a7fc700 1 --2- 172.21.15.94:0/1426810040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efef40037b0 0x7efef4003c20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7eff0000c160 tx=0x7eff00001da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454dd762ddbac34 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:03.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0000ca50 con 0x7efef40037b0 2023-09-30T00:18:03.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0000cbb0 con 0x7efef40037b0 2023-09-30T00:18:03.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.662+0000 7eff09ffb700 1 -- 172.21.15.94:0/1426810040 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00012400 con 0x7efef40037b0 2023-09-30T00:18:03.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.663+0000 7eff115d3700 1 -- 172.21.15.94:0/1426810040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efef40037b0 msgr2=0x7efef4003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.663+0000 7eff115d3700 1 --2- 172.21.15.94:0/1426810040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efef40037b0 0x7efef4003c20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7eff0000c160 tx=0x7eff00001da0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.663+0000 7eff0b7fe700 1 -- 172.21.15.94:0/1426810040 reap_dead start 2023-09-30T00:18:03.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.663+0000 7eff115d3700 1 -- 172.21.15.94:0/1426810040 shutdown_connections 2023-09-30T00:18:03.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.663+0000 7eff115d3700 1 -- 172.21.15.94:0/1426810040 >> 172.21.15.94:0/1426810040 conn(0x7eff0c06af00 msgr2=0x7eff0c0b2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:03.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.663+0000 7fa2bef81700 1 -- 172.21.15.94:0/2441659331 shutdown_connections 2023-09-30T00:18:03.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.663+0000 7eff115d3700 1 -- 172.21.15.94:0/1426810040 shutdown_connections 2023-09-30T00:18:03.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.663+0000 7fa2bef81700 1 -- 172.21.15.94:0/2441659331 wait complete. 2023-09-30T00:18:03.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.663+0000 7eff115d3700 1 -- 172.21.15.94:0/1426810040 wait complete. 2023-09-30T00:18:03.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.664+0000 7eff115d3700 1 Processor -- start 2023-09-30T00:18:03.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff115d3700 1 -- start start 2023-09-30T00:18:03.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7fa2bef81700 1 Processor -- start 2023-09-30T00:18:03.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7fa2bef81700 1 -- start start 2023-09-30T00:18:03.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff115d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff0c06e290 0x7eff0c0a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff115d3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff0c06ce20 0x7eff0c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff115d3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff0c145ca0 0x7eff0c146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff115d3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c06eca0 con 0x7eff0c06ce20 2023-09-30T00:18:03.678 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff115d3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c06e660 con 0x7eff0c06e290 2023-09-30T00:18:03.678 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff115d3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c06dab0 con 0x7eff0c145ca0 2023-09-30T00:18:03.678 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff0a7fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff0c06ce20 0x7eff0c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.678 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff0a7fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff0c06ce20 0x7eff0c06d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50796/0 (socket says 172.21.15.94:50796) 2023-09-30T00:18:03.678 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff0a7fc700 1 -- 172.21.15.94:0/270890542 learned_addr learned my addr 172.21.15.94:0/270890542 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:03.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff0affd700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff0c06e290 0x7eff0c0a6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff0a7fc700 1 -- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff0c145ca0 msgr2=0x7eff0c146110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff0a7fc700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff0c145ca0 0x7eff0c146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff0a7fc700 1 -- 172.21.15.94:0/270890542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff0c06e290 msgr2=0x7eff0c0a6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.665+0000 7eff0a7fc700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff0c06e290 0x7eff0c0a6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7eff0a7fc700 1 -- 172.21.15.94:0/270890542 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c06dd70 con 0x7eff0c06ce20 2023-09-30T00:18:03.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7eff0affd700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff0c06e290 0x7eff0c0a6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:03.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7eff0a7fc700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff0c06ce20 0x7eff0c06d290 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7eff0000c750 tx=0x7eff00015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:03.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2bef81700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b8068a70 0x7fa2b806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2bef81700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8070540 0x7fa2b8096540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2bef81700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2b813ae20 0x7fa2b813b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2b7fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8070540 0x7fa2b8096540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2b7fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8070540 0x7fa2b8096540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39294/0 (socket says 172.21.15.94:39294) 2023-09-30T00:18:03.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2b7fff700 1 -- 172.21.15.94:0/2483960939 learned_addr learned my addr 172.21.15.94:0/2483960939 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:03.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2bef81700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b8134350 con 0x7fa2b8068a70 2023-09-30T00:18:03.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2bcd1d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b8068a70 0x7fa2b806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2bef81700 1 -- 172.21.15.94:0/2483960939 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b8133ff0 con 0x7fa2b813ae20 2023-09-30T00:18:03.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2bef81700 1 -- 172.21.15.94:0/2483960939 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b813c670 con 0x7fa2b8070540 2023-09-30T00:18:03.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7efef2ffd700 1 -- 172.21.15.94:0/270890542 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00001370 con 0x7eff0c06ce20 2023-09-30T00:18:03.683 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7efef2ffd700 1 -- 172.21.15.94:0/270890542 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0001f410 con 0x7eff0c06ce20 2023-09-30T00:18:03.683 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7efef2ffd700 1 -- 172.21.15.94:0/270890542 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0001f7d0 con 0x7eff0c06ce20 2023-09-30T00:18:03.683 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2bd51e700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2b813ae20 0x7fa2b813b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.683 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7eff115d3700 1 -- 172.21.15.94:0/270890542 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0c06d7d0 con 0x7eff0c06ce20 2023-09-30T00:18:03.683 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7eff115d3700 1 -- 172.21.15.94:0/270890542 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0c146810 con 0x7eff0c06ce20 2023-09-30T00:18:03.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2b7fff700 1 -- 172.21.15.94:0/2483960939 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2b813ae20 msgr2=0x7fa2b813b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2b7fff700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2b813ae20 0x7fa2b813b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2b7fff700 1 -- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b8068a70 msgr2=0x7fa2b806ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2b7fff700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b8068a70 0x7fa2b806ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2b7fff700 1 -- 172.21.15.94:0/2483960939 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b0009a20 con 0x7fa2b8070540 2023-09-30T00:18:03.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.666+0000 7fa2bcd1d700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b8068a70 0x7fa2b806ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:03.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.667+0000 7fa2b7fff700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8070540 0x7fa2b8096540 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400c5b0 tx=0x7fa2a4007f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:03.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.667+0000 7fa2b4ff9700 1 -- 172.21.15.94:0/2483960939 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4016070 con 0x7fa2b8070540 2023-09-30T00:18:03.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.667+0000 7fa2b4ff9700 1 -- 172.21.15.94:0/2483960939 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a4006820 con 0x7fa2b8070540 2023-09-30T00:18:03.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.667+0000 7fa2b4ff9700 1 -- 172.21.15.94:0/2483960939 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4006be0 con 0x7fa2b8070540 2023-09-30T00:18:03.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.667+0000 7fa2bef81700 1 -- 172.21.15.94:0/2483960939 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b813d150 con 0x7fa2b8070540 2023-09-30T00:18:03.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.667+0000 7fa2bef81700 1 -- 172.21.15.94:0/2483960939 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b8139850 con 0x7fa2b8070540 2023-09-30T00:18:03.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.667+0000 7eff115d3700 1 -- 172.21.15.94:0/270890542 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0c06eca0 con 0x7eff0c06ce20 2023-09-30T00:18:03.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.668+0000 7fa2bef81700 1 -- 172.21.15.94:0/2483960939 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b8095dc0 con 0x7fa2b8070540 2023-09-30T00:18:03.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.669+0000 7efef2ffd700 1 -- 172.21.15.94:0/270890542 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7eff0001fbc0 con 0x7eff0c06ce20 2023-09-30T00:18:03.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.669+0000 7efef2ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:03.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.670+0000 7efef2ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:03.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.670+0000 7efef2ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:03.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.670+0000 7efef2ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:03.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.670+0000 7efef2ffd700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7efef80987b0 0x7efef809ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.670+0000 7efef2ffd700 1 -- 172.21.15.94:0/270890542 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7eff000c2710 con 0x7eff0c06ce20 2023-09-30T00:18:03.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.670+0000 7fa2b4ff9700 1 -- 172.21.15.94:0/2483960939 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fa2a4007020 con 0x7fa2b8070540 2023-09-30T00:18:03.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.670+0000 7fa2b4ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:03.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.671+0000 7fa2b4ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:03.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.671+0000 7fa2b4ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:03.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.671+0000 7fa2b4ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:03.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.671+0000 7fa2b4ff9700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa29c098780 0x7fa29c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.671+0000 7fa2b4ff9700 1 -- 172.21.15.94:0/2483960939 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fa2a40c0680 con 0x7fa2b8070540 2023-09-30T00:18:03.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.671+0000 7fa2bcd1d700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa29c098780 0x7fa29c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.671+0000 7eff0affd700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7efef80987b0 0x7efef809ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.671+0000 7eff0affd700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7efef80987b0 0x7efef809ac60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efefc001ca0 tx=0x7efefc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:03.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.671+0000 7fa2bcd1d700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa29c098780 0x7fa29c09ac30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac000c00 tx=0x7fa2ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:03.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.674+0000 7efef2ffd700 1 -- 172.21.15.94:0/270890542 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff0008fe50 con 0x7eff0c06ce20 2023-09-30T00:18:03.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.674+0000 7f1d15488700 1 Processor -- start 2023-09-30T00:18:03.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.674+0000 7f1d15488700 1 -- start start 2023-09-30T00:18:03.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.674+0000 7fa2b4ff9700 1 -- 172.21.15.94:0/2483960939 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa2a408ddc0 con 0x7fa2b8070540 2023-09-30T00:18:03.693 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d15488700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 0x7f1d100b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.694 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d15488700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1d1009ca70 0x7f1d1009ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.694 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d15488700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1006ae70 con 0x7f1d1006aaa0 2023-09-30T00:18:03.694 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d15488700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d100b48c0 con 0x7f1d1009ca70 2023-09-30T00:18:03.694 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d15488700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100a45c0 con 0x7f1d100a3ff0 2023-09-30T00:18:03.694 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d0e7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 0x7f1d100b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.694 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d0e7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 0x7f1d100b24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39296/0 (socket says 172.21.15.94:39296) 2023-09-30T00:18:03.694 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d0e7fc700 1 -- 172.21.15.94:0/3456785126 learned_addr learned my addr 172.21.15.94:0/3456785126 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:03.694 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d0effd700 1 --1- 172.21.15.94:0/3456785126 >> v1:172.21.15.121:6789/0 conn(0x7f1d1006aaa0 0x7f1d100b3ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.94:36050/0 (socket says 172.21.15.94:36050) 2023-09-30T00:18:03.694 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d0f7fe700 1 --2- 172.21.15.94:0/3456785126 >> v2:172.21.15.121:3300/0 conn(0x7f1d1009ca70 0x7f1d1009ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.694 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d0e7fc700 1 -- 172.21.15.94:0/3456785126 >> v2:172.21.15.121:3300/0 conn(0x7f1d1009ca70 msgr2=0x7f1d1009ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:03.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d0e7fc700 1 --2- 172.21.15.94:0/3456785126 >> v2:172.21.15.121:3300/0 conn(0x7f1d1009ca70 0x7f1d1009ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d0dffb700 1 -- 172.21.15.94:0/3456785126 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353277856 0 0) 0x7f1d1006ae70 con 0x7f1d1006aaa0 2023-09-30T00:18:03.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d0e7fc700 1 -- 172.21.15.94:0/3456785126 >> v1:172.21.15.121:6789/0 conn(0x7f1d1006aaa0 legacy=0x7f1d100b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d0e7fc700 1 -- 172.21.15.94:0/3456785126 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d100a4880 con 0x7f1d100a3ff0 2023-09-30T00:18:03.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.675+0000 7f1d0e7fc700 1 --2- 172.21.15.94:0/3456785126 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 0x7f1d100b24b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400bc50 tx=0x7f1d0400b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de789178014f735 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:03.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.676+0000 7f1d0dffb700 1 -- 172.21.15.94:0/3456785126 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0400dbf0 con 0x7f1d100a3ff0 2023-09-30T00:18:03.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.676+0000 7f1d0dffb700 1 -- 172.21.15.94:0/3456785126 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0400dd50 con 0x7f1d100a3ff0 2023-09-30T00:18:03.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.676+0000 7f1d0dffb700 1 -- 172.21.15.94:0/3456785126 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d040196b0 con 0x7f1d100a3ff0 2023-09-30T00:18:03.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.677+0000 7f1d15488700 1 -- 172.21.15.94:0/3456785126 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 msgr2=0x7f1d100b24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.677+0000 7f1d15488700 1 --2- 172.21.15.94:0/3456785126 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 0x7f1d100b24b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400bc50 tx=0x7f1d0400b4f0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.677+0000 7f1d15488700 1 -- 172.21.15.94:0/3456785126 shutdown_connections 2023-09-30T00:18:03.695 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.677+0000 7f1d15488700 1 --2- 172.21.15.94:0/3456785126 >> v2:172.21.15.121:3300/0 conn(0x7f1d1009ca70 0x7f1d1009ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.677+0000 7f1d15488700 1 --2- 172.21.15.94:0/3456785126 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 0x7f1d100b24b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.677+0000 7f1d15488700 1 -- 172.21.15.94:0/3456785126 >> 172.21.15.94:0/3456785126 conn(0x7f1d10068730 msgr2=0x7f1d10068b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.677+0000 7f1d15488700 1 -- 172.21.15.94:0/3456785126 shutdown_connections 2023-09-30T00:18:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.677+0000 7f1d15488700 1 -- 172.21.15.94:0/3456785126 wait complete. 2023-09-30T00:18:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.679+0000 7f1d15488700 1 Processor -- start 2023-09-30T00:18:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.679+0000 7f1d15488700 1 -- start start 2023-09-30T00:18:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d15488700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d1006aaa0 0x7f1d100a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d15488700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d1009ca70 0x7f1d1009e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d15488700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 0x7f1d1009e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d15488700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1009c280 con 0x7f1d1009ca70 2023-09-30T00:18:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d15488700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1009ec80 con 0x7f1d1006aaa0 2023-09-30T00:18:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d15488700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1009dd70 con 0x7f1d100a3ff0 2023-09-30T00:18:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0e7fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d1009ca70 0x7f1d1009e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0f7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 0x7f1d1009e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0e7fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d1009ca70 0x7f1d1009e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50808/0 (socket says 172.21.15.94:50808) 2023-09-30T00:18:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0f7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 0x7f1d1009e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39298/0 (socket says 172.21.15.94:39298) 2023-09-30T00:18:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0e7fc700 1 -- 172.21.15.94:0/380376545 learned_addr learned my addr 172.21.15.94:0/380376545 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0effd700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d1006aaa0 0x7f1d100a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0e7fc700 1 -- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 msgr2=0x7f1d1009e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0e7fc700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 0x7f1d1009e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0e7fc700 1 -- 172.21.15.94:0/380376545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d1006aaa0 msgr2=0x7f1d100a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0e7fc700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d1006aaa0 0x7f1d100a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0e7fc700 1 -- 172.21.15.94:0/380376545 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0400b790 con 0x7f1d1009ca70 2023-09-30T00:18:03.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0f7fe700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 0x7f1d1009e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:03.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.680+0000 7f1d0e7fc700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d1009ca70 0x7f1d1009e200 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1d04006fd0 tx=0x7f1d0401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:03.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.681+0000 7f1cf6ffd700 1 -- 172.21.15.94:0/380376545 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04010680 con 0x7f1d1009ca70 2023-09-30T00:18:03.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.681+0000 7f1cf6ffd700 1 -- 172.21.15.94:0/380376545 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d040107e0 con 0x7f1d1009ca70 2023-09-30T00:18:03.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.681+0000 7f1d15488700 1 -- 172.21.15.94:0/380376545 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d10145fc0 con 0x7f1d1009ca70 2023-09-30T00:18:03.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.681+0000 7f1cf6ffd700 1 -- 172.21.15.94:0/380376545 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04010ba0 con 0x7f1d1009ca70 2023-09-30T00:18:03.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.681+0000 7f1d15488700 1 -- 172.21.15.94:0/380376545 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1009f150 con 0x7f1d1009ca70 2023-09-30T00:18:03.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.682+0000 7f1d15488700 1 -- 172.21.15.94:0/380376545 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d1009df20 con 0x7f1d1009ca70 2023-09-30T00:18:03.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.683+0000 7f1cf6ffd700 1 -- 172.21.15.94:0/380376545 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f1d040233f0 con 0x7f1d1009ca70 2023-09-30T00:18:03.698 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.683+0000 7f1cf6ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:03.699 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.684+0000 7f1cf6ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:03.699 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.684+0000 7f1cf6ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:03.699 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.684+0000 7f1cf6ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:03.699 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.684+0000 7f1cf6ffd700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1cfc098780 0x7f1cfc09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:03.699 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.684+0000 7f1d0effd700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1cfc098780 0x7f1cfc09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:03.699 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.684+0000 7f1cf6ffd700 1 -- 172.21.15.94:0/380376545 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f1d04014070 con 0x7f1d1009ca70 2023-09-30T00:18:03.699 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.684+0000 7f1d0effd700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1cfc098780 0x7f1cfc09ac30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d00007fd0 tx=0x7f1d00007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:03.699 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.688+0000 7f1cf6ffd700 1 -- 172.21.15.94:0/380376545 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d1009df20 con 0x7f1d1009ca70 2023-09-30T00:18:03.837 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.837+0000 7fb4a0f06700 1 -- 172.21.15.94:0/1348318651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb49c0b13a0 con 0x7fb49c0a0700 2023-09-30T00:18:03.838 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.837+0000 7fb4837fe700 1 -- 172.21.15.94:0/1348318651 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb494090e90 con 0x7fb49c0a0700 2023-09-30T00:18:03.838 INFO:teuthology.orchestra.run.smithi094.stdout:38654705668 2023-09-30T00:18:03.840 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.839+0000 7fb4a0f06700 10 mgrc shutdown 2023-09-30T00:18:03.840 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.839+0000 7fb4a0f06700 1 -- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb488098700 msgr2=0x7fb48809abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.839+0000 7fb4a0f06700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb488098700 0x7fb48809abb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb48c006fd0 tx=0x7fb48c008040 comp rx=0 tx=0).stop 2023-09-30T00:18:03.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.839+0000 7fb4a0f06700 1 -- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb49c0a0700 msgr2=0x7fb49c09be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.839+0000 7fb4a0f06700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb49c0a0700 0x7fb49c09be50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb49c0a0b00 tx=0x7fb4940015f0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.839+0000 7fb4a0f06700 1 -- 172.21.15.94:0/1348318651 shutdown_connections 2023-09-30T00:18:03.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.840+0000 7fb4a0f06700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb488098700 0x7fb48809abb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.840+0000 7fb4a0f06700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb49c0702f0 0x7fb49c06e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.840+0000 7fb4a0f06700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb49c0a0700 0x7fb49c09be50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.840+0000 7fb4a0f06700 1 --2- 172.21.15.94:0/1348318651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb49c0706c0 0x7fb49c09ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.840+0000 7fb4a0f06700 1 -- 172.21.15.94:0/1348318651 >> 172.21.15.94:0/1348318651 conn(0x7fb49c0a3570 msgr2=0x7fb49c0a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:03.842 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.840+0000 7fb4a0f06700 1 -- 172.21.15.94:0/1348318651 shutdown_connections 2023-09-30T00:18:03.842 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.840+0000 7fb4a0f06700 1 -- 172.21.15.94:0/1348318651 wait complete. 2023-09-30T00:18:03.856 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705668 for osd.0 2023-09-30T00:18:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.877+0000 7eff115d3700 1 -- 172.21.15.94:0/270890542 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7eff0c09b520 con 0x7eff0c06ce20 2023-09-30T00:18:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.877+0000 7efef2ffd700 1 -- 172.21.15.94:0/270890542 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7eff0c09b520 con 0x7eff0c06ce20 2023-09-30T00:18:03.877 INFO:teuthology.orchestra.run.smithi094.stdout:42949672966 2023-09-30T00:18:03.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 10 mgrc shutdown 2023-09-30T00:18:03.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 1 -- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7efef80987b0 msgr2=0x7efef809ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7efef80987b0 0x7efef809ac60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efefc001ca0 tx=0x7efefc009040 comp rx=0 tx=0).stop 2023-09-30T00:18:03.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 1 -- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff0c06ce20 msgr2=0x7eff0c06d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff0c06ce20 0x7eff0c06d290 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7eff0000c750 tx=0x7eff00015000 comp rx=0 tx=0).stop 2023-09-30T00:18:03.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 1 -- 172.21.15.94:0/270890542 shutdown_connections 2023-09-30T00:18:03.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7efef80987b0 0x7efef809ac60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff0c145ca0 0x7eff0c146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff0c06e290 0x7eff0c0a6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 1 --2- 172.21.15.94:0/270890542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff0c06ce20 0x7eff0c06d290 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 1 -- 172.21.15.94:0/270890542 >> 172.21.15.94:0/270890542 conn(0x7eff0c06af00 msgr2=0x7eff0c0a2080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:03.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 1 -- 172.21.15.94:0/270890542 shutdown_connections 2023-09-30T00:18:03.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.879+0000 7eff115d3700 1 -- 172.21.15.94:0/270890542 wait complete. 2023-09-30T00:18:03.882 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.881+0000 7fa2bef81700 1 -- 172.21.15.94:0/2483960939 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa2b806a6a0 con 0x7fa2b8070540 2023-09-30T00:18:03.882 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.881+0000 7fa2b4ff9700 1 -- 172.21.15.94:0/2483960939 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa2a40913e0 con 0x7fa2b8070540 2023-09-30T00:18:03.882 INFO:teuthology.orchestra.run.smithi094.stdout:42949672966 2023-09-30T00:18:03.884 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.883+0000 7fa2bef81700 10 mgrc shutdown 2023-09-30T00:18:03.884 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.883+0000 7fa2bef81700 1 -- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa29c098780 msgr2=0x7fa29c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.884 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.883+0000 7fa2bef81700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa29c098780 0x7fa29c09ac30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac000c00 tx=0x7fa2ac009040 comp rx=0 tx=0).stop 2023-09-30T00:18:03.884 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.883+0000 7fa2bef81700 1 -- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8070540 msgr2=0x7fa2b8096540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.884 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.883+0000 7fa2bef81700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8070540 0x7fa2b8096540 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400c5b0 tx=0x7fa2a4007f20 comp rx=0 tx=0).stop 2023-09-30T00:18:03.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.883+0000 7fa2bef81700 1 -- 172.21.15.94:0/2483960939 shutdown_connections 2023-09-30T00:18:03.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.883+0000 7fa2bef81700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa29c098780 0x7fa29c09ac30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.883+0000 7fa2bef81700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2b813ae20 0x7fa2b813b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.883+0000 7fa2bef81700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b8068a70 0x7fa2b806ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.883+0000 7fa2bef81700 1 --2- 172.21.15.94:0/2483960939 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa2b8070540 0x7fa2b8096540 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.883+0000 7fa2bef81700 1 -- 172.21.15.94:0/2483960939 >> 172.21.15.94:0/2483960939 conn(0x7fa2b80a57c0 msgr2=0x7fa2b80a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:03.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.884+0000 7fa2bef81700 1 -- 172.21.15.94:0/2483960939 shutdown_connections 2023-09-30T00:18:03.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.884+0000 7fa2bef81700 1 -- 172.21.15.94:0/2483960939 wait complete. 2023-09-30T00:18:03.893 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672966 for osd.3 2023-09-30T00:18:03.893 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.893+0000 7f1d15488700 1 -- 172.21.15.94:0/380376545 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1d1009df20 con 0x7f1d1009ca70 2023-09-30T00:18:03.894 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.893+0000 7f1cf6ffd700 1 -- 172.21.15.94:0/380376545 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1d1009df20 con 0x7f1d1009ca70 2023-09-30T00:18:03.894 INFO:teuthology.orchestra.run.smithi094.stdout:42949672966 2023-09-30T00:18:03.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 10 mgrc shutdown 2023-09-30T00:18:03.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 1 -- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1cfc098780 msgr2=0x7f1cfc09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1cfc098780 0x7f1cfc09ac30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d00007fd0 tx=0x7f1d00007d50 comp rx=0 tx=0).stop 2023-09-30T00:18:03.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 1 -- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d1009ca70 msgr2=0x7f1d1009e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:03.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d1009ca70 0x7f1d1009e200 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1d04006fd0 tx=0x7f1d0401b000 comp rx=0 tx=0).stop 2023-09-30T00:18:03.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 1 -- 172.21.15.94:0/380376545 shutdown_connections 2023-09-30T00:18:03.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1cfc098780 0x7f1cfc09ac30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1d100a3ff0 0x7f1d1009e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d1006aaa0 0x7f1d100a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 1 --2- 172.21.15.94:0/380376545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d1009ca70 0x7f1d1009e200 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:03.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 1 -- 172.21.15.94:0/380376545 >> 172.21.15.94:0/380376545 conn(0x7f1d10068730 msgr2=0x7f1d10068b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:03.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 1 -- 172.21.15.94:0/380376545 shutdown_connections 2023-09-30T00:18:03.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:03.895+0000 7f1d15488700 1 -- 172.21.15.94:0/380376545 wait complete. 2023-09-30T00:18:03.898 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672966 for osd.2 2023-09-30T00:18:03.908 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672966 for osd.1 2023-09-30T00:18:04.857 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:18:04.894 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:18:04.899 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:18:04.908 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:18:04.983 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.983+0000 7f5821f5f700 1 Processor -- start 2023-09-30T00:18:04.984 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.983+0000 7f5821f5f700 1 -- start start 2023-09-30T00:18:04.984 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.983+0000 7f5821f5f700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f581c0699b0 0x7f581c09cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:04.985 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.983+0000 7f5821f5f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581c13dba0 con 0x7f581c13cf10 2023-09-30T00:18:04.985 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.983+0000 7f5821f5f700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581c09f290 con 0x7f581c0695e0 2023-09-30T00:18:04.985 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.983+0000 7f5821f5f700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f581c067730 con 0x7f581c0699b0 2023-09-30T00:18:04.986 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.983+0000 7f581bfff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f581c0699b0 0x7f581c09cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:04.986 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.983+0000 7f582175e700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f581c13cf10 0x7f581c13d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46394/0 (socket says 172.21.15.94:46394) 2023-09-30T00:18:04.986 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.983+0000 7f582175e700 1 -- 172.21.15.94:0/3858524768 learned_addr learned my addr 172.21.15.94:0/3858524768 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:04.986 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.984+0000 7f581bfff700 1 -- 172.21.15.94:0/3858524768 >> v1:172.21.15.121:6789/0 conn(0x7f581c0695e0 legacy=0x7f581c09e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:04.986 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.984+0000 7f581bfff700 1 -- 172.21.15.94:0/3858524768 >> v1:172.21.15.94:6789/0 conn(0x7f581c13cf10 legacy=0x7f581c13d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:04.986 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.984+0000 7f581b7fe700 1 -- 172.21.15.94:0/3858524768 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205137286 0 0) 0x7f581c13dba0 con 0x7f581c13cf10 2023-09-30T00:18:04.986 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.984+0000 7f581bfff700 1 -- 172.21.15.94:0/3858524768 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581c09d510 con 0x7f581c0699b0 2023-09-30T00:18:04.986 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.984+0000 7f581bfff700 1 --2- 172.21.15.94:0/3858524768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f581c0699b0 0x7f581c09cfd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f58080098e0 tx=0x7f58080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5e986811581372 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:04.986 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.984+0000 7f581b7fe700 1 -- 172.21.15.94:0/3858524768 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5808009e80 con 0x7f581c0699b0 2023-09-30T00:18:04.986 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.984+0000 7f581b7fe700 1 -- 172.21.15.94:0/3858524768 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580800b860 con 0x7f581c0699b0 2023-09-30T00:18:04.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.984+0000 7f581b7fe700 1 -- 172.21.15.94:0/3858524768 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580800bbe0 con 0x7f581c0699b0 2023-09-30T00:18:04.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.985+0000 7f5821f5f700 1 -- 172.21.15.94:0/3858524768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f581c0699b0 msgr2=0x7f581c09cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:04.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.985+0000 7f5821f5f700 1 --2- 172.21.15.94:0/3858524768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f581c0699b0 0x7f581c09cfd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f58080098e0 tx=0x7f58080091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:04.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.985+0000 7f5821f5f700 1 -- 172.21.15.94:0/3858524768 shutdown_connections 2023-09-30T00:18:04.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.985+0000 7f5821f5f700 1 --2- 172.21.15.94:0/3858524768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f581c0699b0 0x7f581c09cfd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:04.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.985+0000 7f5821f5f700 1 -- 172.21.15.94:0/3858524768 >> 172.21.15.94:0/3858524768 conn(0x7f581c0a3570 msgr2=0x7f581c0a3970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:04.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.985+0000 7f5821f5f700 1 -- 172.21.15.94:0/3858524768 shutdown_connections 2023-09-30T00:18:04.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.985+0000 7f5821f5f700 1 -- 172.21.15.94:0/3858524768 wait complete. 2023-09-30T00:18:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.986+0000 7f5821f5f700 1 Processor -- start 2023-09-30T00:18:04.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.987+0000 7f5821f5f700 1 -- start start 2023-09-30T00:18:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f5821f5f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f581c0695e0 0x7f581c0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f5821f5f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f581c0699b0 0x7f581c0b0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f5821f5f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f581c13cf10 0x7f581c0b1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f5821f5f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c0b2dd0 con 0x7f581c0699b0 2023-09-30T00:18:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f5821f5f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c13f370 con 0x7f581c13cf10 2023-09-30T00:18:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f5821f5f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581c13f610 con 0x7f581c0695e0 2023-09-30T00:18:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f581bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f581c0699b0 0x7f581c0b0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f581bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f581c0699b0 0x7f581c0b0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36190/0 (socket says 172.21.15.94:36190) 2023-09-30T00:18:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f581bfff700 1 -- 172.21.15.94:0/1025022417 learned_addr learned my addr 172.21.15.94:0/1025022417 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f582175e700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f581c13cf10 0x7f581c0b1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:04.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f581bfff700 1 -- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f581c0695e0 msgr2=0x7f581c0b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f581bfff700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f581c0695e0 0x7f581c0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f581bfff700 1 -- 172.21.15.94:0/1025022417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f581c13cf10 msgr2=0x7f581c0b1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f581bfff700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f581c13cf10 0x7f581c0b1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f581bfff700 1 -- 172.21.15.94:0/1025022417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5808009420 con 0x7f581c0699b0 2023-09-30T00:18:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f582175e700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f581c13cf10 0x7f581c0b1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:18:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f581bfff700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f581c0699b0 0x7f581c0b0fd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f581c06bb50 tx=0x7f5808017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.988+0000 7f5818ff9700 1 -- 172.21.15.94:0/1025022417 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580801f070 con 0x7f581c0699b0 2023-09-30T00:18:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.989+0000 7f5818ff9700 1 -- 172.21.15.94:0/1025022417 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5808011420 con 0x7f581c0699b0 2023-09-30T00:18:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.989+0000 7f5821f5f700 1 -- 172.21.15.94:0/1025022417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581c0b0d30 con 0x7f581c0699b0 2023-09-30T00:18:04.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.989+0000 7f5818ff9700 1 -- 172.21.15.94:0/1025022417 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580800bbe0 con 0x7f581c0699b0 2023-09-30T00:18:04.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.989+0000 7f5821f5f700 1 -- 172.21.15.94:0/1025022417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581c13f030 con 0x7f581c0699b0 2023-09-30T00:18:04.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.990+0000 7f5821f5f700 1 -- 172.21.15.94:0/1025022417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581c09d890 con 0x7f581c0699b0 2023-09-30T00:18:04.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.991+0000 7f5818ff9700 1 -- 172.21.15.94:0/1025022417 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5808011a10 con 0x7f581c0699b0 2023-09-30T00:18:04.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.991+0000 7f5818ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:04.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.992+0000 7f5818ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:04.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.992+0000 7f5818ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:04.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.992+0000 7f5818ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:04.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.992+0000 7f5818ff9700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5800098780 0x7f580009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:04.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.992+0000 7f5818ff9700 1 -- 172.21.15.94:0/1025022417 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f58080c1dd0 con 0x7f581c0699b0 2023-09-30T00:18:04.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.992+0000 7f5820f5d700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5800098780 0x7f580009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:04.993 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.993+0000 7f5820f5d700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5800098780 0x7f580009ac30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5810001730 tx=0x7f5810007b40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:04.995 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:04.995+0000 7f5818ff9700 1 -- 172.21.15.94:0/1025022417 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f580808f510 con 0x7f581c0699b0 2023-09-30T00:18:05.025 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.025+0000 7f1788d18700 1 Processor -- start 2023-09-30T00:18:05.026 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.025+0000 7f1788d18700 1 -- start start 2023-09-30T00:18:05.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1788d18700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1784070540 0x7f1784143b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1788d18700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f178409bb30 0x7f178413f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1788d18700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f178409b340 0x7f178413fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1788d18700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1784144090 con 0x7f178409b340 2023-09-30T00:18:05.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1788d18700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1784070910 con 0x7f178409bb30 2023-09-30T00:18:05.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1788d18700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1784099fd0 con 0x7f1784070540 2023-09-30T00:18:05.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1783fff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f178409b340 0x7f178413fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1783fff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f178409b340 0x7f178413fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36198/0 (socket says 172.21.15.94:36198) 2023-09-30T00:18:05.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f17837fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1784070540 0x7f1784143b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1783fff700 1 -- 172.21.15.94:0/900420946 learned_addr learned my addr 172.21.15.94:0/900420946 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:05.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1782ffd700 1 --2- 172.21.15.94:0/900420946 >> v2:172.21.15.121:3300/0 conn(0x7f178409bb30 0x7f178413f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1783fff700 1 -- 172.21.15.94:0/900420946 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1784070540 msgr2=0x7f1784143b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:05.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1783fff700 1 --2- 172.21.15.94:0/900420946 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1784070540 0x7f1784143b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1783fff700 1 -- 172.21.15.94:0/900420946 >> v2:172.21.15.121:3300/0 conn(0x7f178409bb30 msgr2=0x7f178413f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:05.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1783fff700 1 --2- 172.21.15.94:0/900420946 >> v2:172.21.15.121:3300/0 conn(0x7f178409bb30 0x7f178413f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1783fff700 1 -- 172.21.15.94:0/900420946 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17841442a0 con 0x7f178409b340 2023-09-30T00:18:05.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f17837fe700 1 --2- 172.21.15.94:0/900420946 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1784070540 0x7f1784143b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:05.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.026+0000 7f1783fff700 1 --2- 172.21.15.94:0/900420946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f178409b340 0x7f178413fdd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f178406b870 tx=0x7f177c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad20fc3aec9dd8d5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.027+0000 7f17827fc700 1 -- 172.21.15.94:0/900420946 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c00e070 con 0x7f178409b340 2023-09-30T00:18:05.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.027+0000 7f17827fc700 1 -- 172.21.15.94:0/900420946 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177c00b970 con 0x7f178409b340 2023-09-30T00:18:05.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.027+0000 7f17827fc700 1 -- 172.21.15.94:0/900420946 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c0092e0 con 0x7f178409b340 2023-09-30T00:18:05.030 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.027+0000 7f5ef97c6700 1 Processor -- start 2023-09-30T00:18:05.030 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.027+0000 7f5ef97c6700 1 -- start start 2023-09-30T00:18:05.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.027+0000 7f1788d18700 1 -- 172.21.15.94:0/900420946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f178409b340 msgr2=0x7f178413fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.027+0000 7f1788d18700 1 --2- 172.21.15.94:0/900420946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f178409b340 0x7f178413fdd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f178406b870 tx=0x7f177c009840 comp rx=0 tx=0).stop 2023-09-30T00:18:05.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.027+0000 7f1788d18700 1 -- 172.21.15.94:0/900420946 shutdown_connections 2023-09-30T00:18:05.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.027+0000 7f1788d18700 1 --2- 172.21.15.94:0/900420946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f178409b340 0x7f178413fdd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.027+0000 7f1788d18700 1 --2- 172.21.15.94:0/900420946 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1784070540 0x7f1784143b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.027+0000 7f1788d18700 1 --2- 172.21.15.94:0/900420946 >> v2:172.21.15.121:3300/0 conn(0x7f178409bb30 0x7f178413f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.027+0000 7f1788d18700 1 -- 172.21.15.94:0/900420946 >> 172.21.15.94:0/900420946 conn(0x7f17840a3570 msgr2=0x7f17840a3970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:05.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f1788d18700 1 -- 172.21.15.94:0/900420946 shutdown_connections 2023-09-30T00:18:05.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f1788d18700 1 -- 172.21.15.94:0/900420946 wait complete. 2023-09-30T00:18:05.033 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef97c6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef409da80 0x7f5ef409f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.033 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef97c6700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5ef409de50 0x7f5ef4147e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.033 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef97c6700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef413d430 con 0x7f5ef413de90 2023-09-30T00:18:05.033 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef97c6700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef413d670 con 0x7f5ef409de50 2023-09-30T00:18:05.034 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef97c6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef413d8d0 con 0x7f5ef409da80 2023-09-30T00:18:05.034 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef2ffd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5ef413de90 0x7f5ef40702f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46404/0 (socket says 172.21.15.94:46404) 2023-09-30T00:18:05.034 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef2ffd700 1 -- 172.21.15.94:0/2201998019 learned_addr learned my addr 172.21.15.94:0/2201998019 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:05.034 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef27fc700 1 --2- 172.21.15.94:0/2201998019 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef409da80 0x7f5ef409f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.034 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef37fe700 1 --2- 172.21.15.94:0/2201998019 >> v2:172.21.15.121:3300/0 conn(0x7f5ef409de50 0x7f5ef4147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.034 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef1ffb700 1 -- 172.21.15.94:0/2201998019 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726341055 0 0) 0x7f5ee4001290 con 0x7f5ef413de90 2023-09-30T00:18:05.034 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef1ffb700 1 -- 172.21.15.94:0/2201998019 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5edc003570 con 0x7f5ef413de90 2023-09-30T00:18:05.034 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef27fc700 1 -- 172.21.15.94:0/2201998019 >> v2:172.21.15.121:3300/0 conn(0x7f5ef409de50 msgr2=0x7f5ef4147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:05.034 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef27fc700 1 --2- 172.21.15.94:0/2201998019 >> v2:172.21.15.121:3300/0 conn(0x7f5ef409de50 0x7f5ef4147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.034 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef27fc700 1 -- 172.21.15.94:0/2201998019 >> v1:172.21.15.94:6789/0 conn(0x7f5ef413de90 legacy=0x7f5ef40702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.034 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef27fc700 1 -- 172.21.15.94:0/2201998019 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef413db70 con 0x7f5ef409da80 2023-09-30T00:18:05.035 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.028+0000 7f5ef27fc700 1 --2- 172.21.15.94:0/2201998019 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef409da80 0x7f5ef409f0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800bb90 tx=0x7f5ee800b430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39a56097ca3c725e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.035 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.029+0000 7f5ef1ffb700 1 -- 172.21.15.94:0/2201998019 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee800da70 con 0x7f5ef409da80 2023-09-30T00:18:05.035 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.029+0000 7f5ef1ffb700 1 -- 172.21.15.94:0/2201998019 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee800dbd0 con 0x7f5ef409da80 2023-09-30T00:18:05.035 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.029+0000 7f5ef1ffb700 1 -- 172.21.15.94:0/2201998019 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8019480 con 0x7f5ef409da80 2023-09-30T00:18:05.035 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.029+0000 7f1788d18700 1 Processor -- start 2023-09-30T00:18:05.035 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.029+0000 7f1788d18700 1 -- start start 2023-09-30T00:18:05.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.029+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2201998019 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef409da80 msgr2=0x7f5ef409f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.029+0000 7f5ef97c6700 1 --2- 172.21.15.94:0/2201998019 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef409da80 0x7f5ef409f0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800bb90 tx=0x7f5ee800b430 comp rx=0 tx=0).stop 2023-09-30T00:18:05.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.029+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2201998019 shutdown_connections 2023-09-30T00:18:05.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.029+0000 7f5ef97c6700 1 --2- 172.21.15.94:0/2201998019 >> v2:172.21.15.121:3300/0 conn(0x7f5ef409de50 0x7f5ef4147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.029+0000 7f5ef97c6700 1 --2- 172.21.15.94:0/2201998019 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef409da80 0x7f5ef409f0d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.029+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2201998019 >> 172.21.15.94:0/2201998019 conn(0x7f5ef40a57c0 msgr2=0x7f5ef40a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:05.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2201998019 shutdown_connections 2023-09-30T00:18:05.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2201998019 wait complete. 2023-09-30T00:18:05.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1788d18700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1784070540 0x7f17840a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1788d18700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f178409b340 0x7f178413c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1788d18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f178409bb30 0x7f178413aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1788d18700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178406c260 con 0x7f178409b340 2023-09-30T00:18:05.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1788d18700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17840a1fd0 con 0x7f178409bb30 2023-09-30T00:18:05.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1788d18700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17840b1820 con 0x7f1784070540 2023-09-30T00:18:05.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1782ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f178409b340 0x7f178413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1782ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f178409b340 0x7f178413c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36204/0 (socket says 172.21.15.94:36204) 2023-09-30T00:18:05.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1782ffd700 1 -- 172.21.15.94:0/934181386 learned_addr learned my addr 172.21.15.94:0/934181386 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:05.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f17837fe700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1784070540 0x7f17840a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1783fff700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f178409bb30 0x7f178413aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1782ffd700 1 -- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1784070540 msgr2=0x7f17840a29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1782ffd700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1784070540 0x7f17840a29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1782ffd700 1 -- 172.21.15.94:0/934181386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f178409bb30 msgr2=0x7f178413aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1782ffd700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f178409bb30 0x7f178413aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1782ffd700 1 -- 172.21.15.94:0/934181386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1774009420 con 0x7f178409b340 2023-09-30T00:18:05.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1782ffd700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f178409b340 0x7f178413c2f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f177800e960 tx=0x7f17780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f176b7fe700 1 -- 172.21.15.94:0/934181386 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778013680 con 0x7f178409b340 2023-09-30T00:18:05.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f176b7fe700 1 -- 172.21.15.94:0/934181386 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17780137e0 con 0x7f178409b340 2023-09-30T00:18:05.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1788d18700 1 -- 172.21.15.94:0/934181386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f177c009ae0 con 0x7f178409b340 2023-09-30T00:18:05.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f176b7fe700 1 -- 172.21.15.94:0/934181386 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778013b80 con 0x7f178409b340 2023-09-30T00:18:05.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.030+0000 7f1788d18700 1 -- 172.21.15.94:0/934181386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178413c830 con 0x7f178409b340 2023-09-30T00:18:05.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.031+0000 7f5ef97c6700 1 Processor -- start 2023-09-30T00:18:05.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.031+0000 7f5ef97c6700 1 -- start start 2023-09-30T00:18:05.041 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.031+0000 7f1788d18700 1 -- 172.21.15.94:0/934181386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178413c010 con 0x7f178409b340 2023-09-30T00:18:05.041 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.032+0000 7f5ef97c6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ef409da80 0x7f5ef409c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.041 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.032+0000 7f5ef97c6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ef409de50 0x7f5ef406e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.041 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.032+0000 7f5ef97c6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef413de90 0x7f5ef414fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.032+0000 7f5ef97c6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef4150bf0 con 0x7f5ef409de50 2023-09-30T00:18:05.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.032+0000 7f5ef97c6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef4150e50 con 0x7f5ef409da80 2023-09-30T00:18:05.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.032+0000 7f5ef97c6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef4151110 con 0x7f5ef413de90 2023-09-30T00:18:05.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.032+0000 7f5ef37fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef413de90 0x7f5ef414fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.032+0000 7f5ef37fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef413de90 0x7f5ef414fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:37280/0 (socket says 172.21.15.94:37280) 2023-09-30T00:18:05.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.032+0000 7f5ef37fe700 1 -- 172.21.15.94:0/2235198463 learned_addr learned my addr 172.21.15.94:0/2235198463 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:05.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.032+0000 7f5ef2ffd700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ef409da80 0x7f5ef409c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.032+0000 7f5ef27fc700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ef409de50 0x7f5ef406e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f1783fff700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f178409bb30 0x7f178413aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:18:05.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f176b7fe700 1 -- 172.21.15.94:0/934181386 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f177800c490 con 0x7f178409b340 2023-09-30T00:18:05.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f176b7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:05.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f5ef37fe700 1 -- 172.21.15.94:0/2235198463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ef409da80 msgr2=0x7f5ef409c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f5ef37fe700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ef409da80 0x7f5ef409c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f5ef37fe700 1 -- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ef409de50 msgr2=0x7f5ef406e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f5ef37fe700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ef409de50 0x7f5ef406e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f5ef37fe700 1 -- 172.21.15.94:0/2235198463 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee800b6d0 con 0x7f5ef413de90 2023-09-30T00:18:05.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f5ef27fc700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ef409de50 0x7f5ef406e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:05.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f5ef37fe700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef413de90 0x7f5ef414fed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009df0 tx=0x7f5eec00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f5edaffd700 1 -- 172.21.15.94:0/2235198463 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec00cb50 con 0x7f5ef413de90 2023-09-30T00:18:05.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f5edaffd700 1 -- 172.21.15.94:0/2235198463 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eec00f040 con 0x7f5ef413de90 2023-09-30T00:18:05.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2235198463 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef409c8d0 con 0x7f5ef413de90 2023-09-30T00:18:05.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f176b7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:05.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f176b7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:05.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f176b7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:05.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f176b7fe700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1770098780 0x7f177009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.034+0000 7f176b7fe700 1 -- 172.21.15.94:0/934181386 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f17780c0b40 con 0x7f178409b340 2023-09-30T00:18:05.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.034+0000 7f17837fe700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1770098780 0x7f177009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.034+0000 7f17837fe700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1770098780 0x7f177009ac30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1774009f40 tx=0x7f17740091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f5edaffd700 1 -- 172.21.15.94:0/2235198463 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec014480 con 0x7f5ef413de90 2023-09-30T00:18:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.033+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2235198463 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef406ed20 con 0x7f5ef413de90 2023-09-30T00:18:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.035+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2235198463 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef406a6a0 con 0x7f5ef413de90 2023-09-30T00:18:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.036+0000 7f5edaffd700 1 -- 172.21.15.94:0/2235198463 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5eec0147f0 con 0x7f5ef413de90 2023-09-30T00:18:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.036+0000 7f5edaffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.037+0000 7f5edaffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.037+0000 7f5edaffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.037+0000 7f5edaffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.037+0000 7f5edaffd700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5ee0098700 0x7f5ee009abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.037+0000 7f5ef2ffd700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5ee0098700 0x7f5ee009abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.037+0000 7f5edaffd700 1 -- 172.21.15.94:0/2235198463 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f5eec0c44f0 con 0x7f5ef413de90 2023-09-30T00:18:05.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.037+0000 7f5ef2ffd700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5ee0098700 0x7f5ee009abb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40013f0 tx=0x7f5ee4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.039+0000 7f176b7fe700 1 -- 172.21.15.94:0/934181386 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f177808e2a0 con 0x7f178409b340 2023-09-30T00:18:05.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.042+0000 7f5edaffd700 1 -- 172.21.15.94:0/2235198463 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5eec08daf0 con 0x7f5ef413de90 2023-09-30T00:18:05.070 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.070+0000 7f3cd4129700 1 Processor -- start 2023-09-30T00:18:05.071 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.070+0000 7f3cd4129700 1 -- start start 2023-09-30T00:18:05.072 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd4129700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ccc0a46c0 0x7f3ccc068c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.072 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd4129700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3ccc06a460 0x7f3ccc0b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.072 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd4129700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3ccc0691b0 0x7f3ccc0b5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.072 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd4129700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ccc0a57f0 con 0x7f3ccc0691b0 2023-09-30T00:18:05.072 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd4129700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ccc0b5bc0 con 0x7f3ccc06a460 2023-09-30T00:18:05.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd4129700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc069760 con 0x7f3ccc0a46c0 2023-09-30T00:18:05.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd1ec5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ccc0a46c0 0x7f3ccc068c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd1ec5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ccc0a46c0 0x7f3ccc068c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:37284/0 (socket says 172.21.15.94:37284) 2023-09-30T00:18:05.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd1ec5700 1 -- 172.21.15.94:0/3904968621 learned_addr learned my addr 172.21.15.94:0/3904968621 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:05.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd26c6700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3ccc0691b0 0x7f3ccc0b5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd16c4700 1 --2- 172.21.15.94:0/3904968621 >> v2:172.21.15.121:3300/0 conn(0x7f3ccc06a460 0x7f3ccc0b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd26c6700 1 -- 172.21.15.94:0/3904968621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ccc0a46c0 msgr2=0x7f3ccc068c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:05.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd26c6700 1 --2- 172.21.15.94:0/3904968621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ccc0a46c0 0x7f3ccc068c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd26c6700 1 -- 172.21.15.94:0/3904968621 >> v2:172.21.15.121:3300/0 conn(0x7f3ccc06a460 msgr2=0x7f3ccc0b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:05.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd26c6700 1 --2- 172.21.15.94:0/3904968621 >> v2:172.21.15.121:3300/0 conn(0x7f3ccc06a460 0x7f3ccc0b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd26c6700 1 -- 172.21.15.94:0/3904968621 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc0a4a90 con 0x7f3ccc0691b0 2023-09-30T00:18:05.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd1ec5700 1 --2- 172.21.15.94:0/3904968621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ccc0a46c0 0x7f3ccc068c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:05.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.071+0000 7f3cd26c6700 1 --2- 172.21.15.94:0/3904968621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ccc0691b0 0x7f3ccc0b5680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc06d790 tx=0x7f3cc4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc8a45e461bf83d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.072+0000 7f3cd0ec3700 1 -- 172.21.15.94:0/3904968621 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc400e070 con 0x7f3ccc0691b0 2023-09-30T00:18:05.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.072+0000 7f3cd0ec3700 1 -- 172.21.15.94:0/3904968621 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc400bbb0 con 0x7f3ccc0691b0 2023-09-30T00:18:05.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.072+0000 7f3cd0ec3700 1 -- 172.21.15.94:0/3904968621 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4012430 con 0x7f3ccc0691b0 2023-09-30T00:18:05.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.072+0000 7f3cd4129700 1 -- 172.21.15.94:0/3904968621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ccc0691b0 msgr2=0x7f3ccc0b5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.072+0000 7f3cd4129700 1 --2- 172.21.15.94:0/3904968621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ccc0691b0 0x7f3ccc0b5680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc06d790 tx=0x7f3cc4009780 comp rx=0 tx=0).stop 2023-09-30T00:18:05.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.072+0000 7f3cd4129700 1 -- 172.21.15.94:0/3904968621 shutdown_connections 2023-09-30T00:18:05.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.072+0000 7f3cd4129700 1 --2- 172.21.15.94:0/3904968621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ccc0691b0 0x7f3ccc0b5680 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.072+0000 7f3cd4129700 1 --2- 172.21.15.94:0/3904968621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ccc0a46c0 0x7f3ccc068c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.072+0000 7f3cd4129700 1 --2- 172.21.15.94:0/3904968621 >> v2:172.21.15.121:3300/0 conn(0x7f3ccc06a460 0x7f3ccc0b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.072+0000 7f3cd4129700 1 -- 172.21.15.94:0/3904968621 >> 172.21.15.94:0/3904968621 conn(0x7f3ccc06cbf0 msgr2=0x7f3ccc06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:05.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.072+0000 7f3cd4129700 1 -- 172.21.15.94:0/3904968621 shutdown_connections 2023-09-30T00:18:05.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.072+0000 7f3cd4129700 1 -- 172.21.15.94:0/3904968621 wait complete. 2023-09-30T00:18:05.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.073+0000 7f3cd4129700 1 Processor -- start 2023-09-30T00:18:05.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.073+0000 7f3cd4129700 1 -- start start 2023-09-30T00:18:05.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd4129700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ccc0691b0 0x7f3ccc0b1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd4129700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ccc06a460 0x7f3ccc11d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd4129700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ccc0a46c0 0x7f3ccc13ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd4129700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc1394f0 con 0x7f3ccc06a460 2023-09-30T00:18:05.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd4129700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0ae810 con 0x7f3ccc0691b0 2023-09-30T00:18:05.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd4129700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc140f90 con 0x7f3ccc0a46c0 2023-09-30T00:18:05.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd16c4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ccc06a460 0x7f3ccc11d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd16c4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ccc06a460 0x7f3ccc11d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36232/0 (socket says 172.21.15.94:36232) 2023-09-30T00:18:05.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd16c4700 1 -- 172.21.15.94:0/3791132475 learned_addr learned my addr 172.21.15.94:0/3791132475 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:05.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd1ec5700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ccc0691b0 0x7f3ccc0b1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd26c6700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ccc0a46c0 0x7f3ccc13ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd16c4700 1 -- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ccc0a46c0 msgr2=0x7f3ccc13ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd16c4700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ccc0a46c0 0x7f3ccc13ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd16c4700 1 -- 172.21.15.94:0/3791132475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ccc0691b0 msgr2=0x7f3ccc0b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd16c4700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ccc0691b0 0x7f3ccc0b1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd16c4700 1 -- 172.21.15.94:0/3791132475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc80094f0 con 0x7f3ccc06a460 2023-09-30T00:18:05.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.074+0000 7f3cd16c4700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ccc06a460 0x7f3ccc11d230 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00d890 tx=0x7f3cbc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.075+0000 7f3cc1ffb700 1 -- 172.21.15.94:0/3791132475 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc0125f0 con 0x7f3ccc06a460 2023-09-30T00:18:05.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.075+0000 7f3cc1ffb700 1 -- 172.21.15.94:0/3791132475 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cbc012750 con 0x7f3ccc06a460 2023-09-30T00:18:05.079 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.075+0000 7f3cd4129700 1 -- 172.21.15.94:0/3791132475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc4009a20 con 0x7f3ccc06a460 2023-09-30T00:18:05.080 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.075+0000 7f3cc1ffb700 1 -- 172.21.15.94:0/3791132475 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc012af0 con 0x7f3ccc06a460 2023-09-30T00:18:05.080 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.075+0000 7f3cd4129700 1 -- 172.21.15.94:0/3791132475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ccc11d770 con 0x7f3ccc06a460 2023-09-30T00:18:05.080 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.076+0000 7f3cd4129700 1 -- 172.21.15.94:0/3791132475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ccc06fbd0 con 0x7f3ccc06a460 2023-09-30T00:18:05.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.077+0000 7f3cc1ffb700 1 -- 172.21.15.94:0/3791132475 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f3cbc01e460 con 0x7f3ccc06a460 2023-09-30T00:18:05.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.077+0000 7f3cc1ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:05.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.078+0000 7f3cc1ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:05.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.078+0000 7f3cc1ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:05.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.078+0000 7f3cc1ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:05.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.078+0000 7f3cc1ffb700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3cb0098780 0x7f3cb009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.078+0000 7f3cd1ec5700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3cb0098780 0x7f3cb009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.078+0000 7f3cc1ffb700 1 -- 172.21.15.94:0/3791132475 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f3cbc0c1460 con 0x7f3ccc06a460 2023-09-30T00:18:05.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.078+0000 7f3cd1ec5700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3cb0098780 0x7f3cb009ac30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8006d20 tx=0x7f3cc8009250 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.081+0000 7f3cc1ffb700 1 -- 172.21.15.94:0/3791132475 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3cbc08eaf0 con 0x7f3ccc06a460 2023-09-30T00:18:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.229+0000 7f5821f5f700 1 -- 172.21.15.94:0/1025022417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f581c04c380 con 0x7f581c0699b0 2023-09-30T00:18:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.229+0000 7f5818ff9700 1 -- 172.21.15.94:0/1025022417 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5808028090 con 0x7f581c0699b0 2023-09-30T00:18:05.231 INFO:teuthology.orchestra.run.smithi094.stdout:38654705670 2023-09-30T00:18:05.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.232+0000 7f5821f5f700 10 mgrc shutdown 2023-09-30T00:18:05.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.232+0000 7f5821f5f700 1 -- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5800098780 msgr2=0x7f580009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.232+0000 7f5821f5f700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5800098780 0x7f580009ac30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5810001730 tx=0x7f5810007b40 comp rx=0 tx=0).stop 2023-09-30T00:18:05.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.232+0000 7f5821f5f700 1 -- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f581c0699b0 msgr2=0x7f581c0b0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.232+0000 7f5821f5f700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f581c0699b0 0x7f581c0b0fd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f581c06bb50 tx=0x7f5808017000 comp rx=0 tx=0).stop 2023-09-30T00:18:05.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.232+0000 7f5821f5f700 1 -- 172.21.15.94:0/1025022417 shutdown_connections 2023-09-30T00:18:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.232+0000 7f5821f5f700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5800098780 0x7f580009ac30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.232+0000 7f5821f5f700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f581c0695e0 0x7f581c0b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.232+0000 7f5821f5f700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f581c13cf10 0x7f581c0b1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.232+0000 7f5821f5f700 1 --2- 172.21.15.94:0/1025022417 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f581c0699b0 0x7f581c0b0fd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.232+0000 7f5821f5f700 1 -- 172.21.15.94:0/1025022417 >> 172.21.15.94:0/1025022417 conn(0x7f581c0a3570 msgr2=0x7f581c0a3940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.233+0000 7f5821f5f700 1 -- 172.21.15.94:0/1025022417 shutdown_connections 2023-09-30T00:18:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.233+0000 7f5821f5f700 1 -- 172.21.15.94:0/1025022417 wait complete. 2023-09-30T00:18:05.236 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.236+0000 7f1788d18700 1 -- 172.21.15.94:0/934181386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f178413c010 con 0x7f178409b340 2023-09-30T00:18:05.236 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.236+0000 7f176b7fe700 1 -- 172.21.15.94:0/934181386 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f17780918c0 con 0x7f178409b340 2023-09-30T00:18:05.237 INFO:teuthology.orchestra.run.smithi094.stdout:42949672968 2023-09-30T00:18:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 10 mgrc shutdown 2023-09-30T00:18:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 1 -- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1770098780 msgr2=0x7f177009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1770098780 0x7f177009ac30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1774009f40 tx=0x7f17740091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 1 -- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f178409b340 msgr2=0x7f178413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f178409b340 0x7f178413c2f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f177800e960 tx=0x7f17780091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 1 -- 172.21.15.94:0/934181386 shutdown_connections 2023-09-30T00:18:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1770098780 0x7f177009ac30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1784070540 0x7f17840a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f178409bb30 0x7f178413aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 1 --2- 172.21.15.94:0/934181386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f178409b340 0x7f178413c2f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 1 -- 172.21.15.94:0/934181386 >> 172.21.15.94:0/934181386 conn(0x7f17840a3570 msgr2=0x7f1784143580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 1 -- 172.21.15.94:0/934181386 shutdown_connections 2023-09-30T00:18:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.239+0000 7f1788d18700 1 -- 172.21.15.94:0/934181386 wait complete. 2023-09-30T00:18:05.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.0 2023-09-30T00:18:05.246 DEBUG:teuthology.parallel:result is None 2023-09-30T00:18:05.257 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672968 for osd.2 2023-09-30T00:18:05.257 DEBUG:teuthology.parallel:result is None 2023-09-30T00:18:05.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.277+0000 7f3cd4129700 1 -- 172.21.15.94:0/3791132475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3ccc06ecb0 con 0x7f3ccc06a460 2023-09-30T00:18:05.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.277+0000 7f3cc1ffb700 1 -- 172.21.15.94:0/3791132475 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3cbc092110 con 0x7f3ccc06a460 2023-09-30T00:18:05.278 INFO:teuthology.orchestra.run.smithi094.stdout:42949672968 2023-09-30T00:18:05.280 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 10 mgrc shutdown 2023-09-30T00:18:05.280 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 1 -- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3cb0098780 msgr2=0x7f3cb009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.280 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3cb0098780 0x7f3cb009ac30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8006d20 tx=0x7f3cc8009250 comp rx=0 tx=0).stop 2023-09-30T00:18:05.280 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 1 -- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ccc06a460 msgr2=0x7f3ccc11d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.280 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ccc06a460 0x7f3ccc11d230 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00d890 tx=0x7f3cbc008040 comp rx=0 tx=0).stop 2023-09-30T00:18:05.280 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 1 -- 172.21.15.94:0/3791132475 shutdown_connections 2023-09-30T00:18:05.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3cb0098780 0x7f3cb009ac30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ccc0a46c0 0x7f3ccc13ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ccc0691b0 0x7f3ccc0b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 1 --2- 172.21.15.94:0/3791132475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ccc06a460 0x7f3ccc11d230 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 1 -- 172.21.15.94:0/3791132475 >> 172.21.15.94:0/3791132475 conn(0x7f3ccc06cbf0 msgr2=0x7f3ccc0b4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:05.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 1 -- 172.21.15.94:0/3791132475 shutdown_connections 2023-09-30T00:18:05.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.279+0000 7f3cd4129700 1 -- 172.21.15.94:0/3791132475 wait complete. 2023-09-30T00:18:05.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.284+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2235198463 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5ef406cfe0 con 0x7f5ef413de90 2023-09-30T00:18:05.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.284+0000 7f5edaffd700 1 -- 172.21.15.94:0/2235198463 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5eec091110 con 0x7f5ef413de90 2023-09-30T00:18:05.284 INFO:teuthology.orchestra.run.smithi094.stdout:42949672968 2023-09-30T00:18:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 10 mgrc shutdown 2023-09-30T00:18:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5ee0098700 msgr2=0x7f5ee009abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5ee0098700 0x7f5ee009abb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40013f0 tx=0x7f5ee4009040 comp rx=0 tx=0).stop 2023-09-30T00:18:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef413de90 msgr2=0x7f5ef414fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef413de90 0x7f5ef414fed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009df0 tx=0x7f5eec00b040 comp rx=0 tx=0).stop 2023-09-30T00:18:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2235198463 shutdown_connections 2023-09-30T00:18:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5ee0098700 0x7f5ee009abb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5ef413de90 0x7f5ef414fed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ef409da80 0x7f5ef409c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 1 --2- 172.21.15.94:0/2235198463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5ef409de50 0x7f5ef406e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2235198463 >> 172.21.15.94:0/2235198463 conn(0x7f5ef40a57c0 msgr2=0x7f5ef40b14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2235198463 shutdown_connections 2023-09-30T00:18:05.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.286+0000 7f5ef97c6700 1 -- 172.21.15.94:0/2235198463 wait complete. 2023-09-30T00:18:05.292 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672968 for osd.1 2023-09-30T00:18:05.292 DEBUG:teuthology.parallel:result is None 2023-09-30T00:18:05.299 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672967 got 42949672968 for osd.3 2023-09-30T00:18:05.299 DEBUG:teuthology.parallel:result is None 2023-09-30T00:18:05.299 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:18:05.299 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:18:05.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84c1303700 1 Processor -- start 2023-09-30T00:18:05.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84c1303700 1 -- start start 2023-09-30T00:18:05.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84c1303700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84bc0b7160 0x7f84bc0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84c1303700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc13fd70 0x7f84bc1401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84c1303700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc0a1310 con 0x7f84bc070c50 2023-09-30T00:18:05.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84c1303700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84bc071020 con 0x7f84bc0b7160 2023-09-30T00:18:05.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84c1303700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc0b76a0 con 0x7f84bc13fd70 2023-09-30T00:18:05.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84ba7fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84bc0b7160 0x7f84bc0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84bb7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc13fd70 0x7f84bc1401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.455 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84ba7fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84bc0b7160 0x7f84bc0a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36234/0 (socket says 172.21.15.94:36234) 2023-09-30T00:18:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84baffd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f84bc070c50 0x7f84bc0a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46412/0 (socket says 172.21.15.94:46412) 2023-09-30T00:18:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84ba7fc700 1 -- 172.21.15.94:0/3438333411 learned_addr learned my addr 172.21.15.94:0/3438333411 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.452+0000 7f84bb7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc13fd70 0x7f84bc1401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:37302/0 (socket says 172.21.15.94:37302) 2023-09-30T00:18:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84bb7fe700 1 -- 172.21.15.94:0/3438333411 >> v2:172.21.15.94:3300/0 conn(0x7f84bc0b7160 msgr2=0x7f84bc0a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84bb7fe700 1 --2- 172.21.15.94:0/3438333411 >> v2:172.21.15.94:3300/0 conn(0x7f84bc0b7160 0x7f84bc0a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84bb7fe700 1 -- 172.21.15.94:0/3438333411 >> v1:172.21.15.94:6789/0 conn(0x7f84bc070c50 legacy=0x7f84bc0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84b9ffb700 1 -- 172.21.15.94:0/3438333411 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195127794 0 0) 0x7f84ac001490 con 0x7f84bc070c50 2023-09-30T00:18:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84bb7fe700 1 -- 172.21.15.94:0/3438333411 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc0b7940 con 0x7f84bc13fd70 2023-09-30T00:18:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84ba7fc700 1 --2- 172.21.15.94:0/3438333411 >> v2:172.21.15.94:3300/0 conn(0x7f84bc0b7160 0x7f84bc0a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84bb7fe700 1 --2- 172.21.15.94:0/3438333411 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc13fd70 0x7f84bc1401d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f84b40098e0 tx=0x7f84b40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d3e4c7ff529554d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84b9ffb700 1 -- 172.21.15.94:0/3438333411 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b400b6d0 con 0x7f84bc13fd70 2023-09-30T00:18:05.456 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84b9ffb700 1 -- 172.21.15.94:0/3438333411 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b400b830 con 0x7f84bc13fd70 2023-09-30T00:18:05.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84b9ffb700 1 -- 172.21.15.94:0/3438333411 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b400bbd0 con 0x7f84bc13fd70 2023-09-30T00:18:05.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84c1303700 1 -- 172.21.15.94:0/3438333411 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc13fd70 msgr2=0x7f84bc1401d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84c1303700 1 --2- 172.21.15.94:0/3438333411 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc13fd70 0x7f84bc1401d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f84b40098e0 tx=0x7f84b40091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84c1303700 1 -- 172.21.15.94:0/3438333411 shutdown_connections 2023-09-30T00:18:05.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84c1303700 1 --2- 172.21.15.94:0/3438333411 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc13fd70 0x7f84bc1401d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84c1303700 1 --2- 172.21.15.94:0/3438333411 >> v2:172.21.15.94:3300/0 conn(0x7f84bc0b7160 0x7f84bc0a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.453+0000 7f84c1303700 1 -- 172.21.15.94:0/3438333411 >> 172.21.15.94:0/3438333411 conn(0x7f84bc06a090 msgr2=0x7f84bc0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:05.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.454+0000 7f84c1303700 1 -- 172.21.15.94:0/3438333411 shutdown_connections 2023-09-30T00:18:05.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.454+0000 7f84c1303700 1 -- 172.21.15.94:0/3438333411 wait complete. 2023-09-30T00:18:05.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84c1303700 1 Processor -- start 2023-09-30T00:18:05.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84c1303700 1 -- start start 2023-09-30T00:18:05.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84c1303700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc070c50 0x7f84bc0a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84c1303700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84bc0b7160 0x7f84bc13b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84c1303700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84bc13fd70 0x7f84bc139ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84c1303700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc0a4180 con 0x7f84bc0b7160 2023-09-30T00:18:05.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84c1303700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc13beb0 con 0x7f84bc13fd70 2023-09-30T00:18:05.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84c1303700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc143180 con 0x7f84bc070c50 2023-09-30T00:18:05.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84baffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc070c50 0x7f84bc0a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84ba7fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84bc0b7160 0x7f84bc13b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84baffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc070c50 0x7f84bc0a3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:37310/0 (socket says 172.21.15.94:37310) 2023-09-30T00:18:05.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84baffd700 1 -- 172.21.15.94:0/3314351850 learned_addr learned my addr 172.21.15.94:0/3314351850 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:05.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84bb7fe700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84bc13fd70 0x7f84bc139ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.461 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84baffd700 1 -- 172.21.15.94:0/3314351850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84bc13fd70 msgr2=0x7f84bc139ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.461 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84baffd700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84bc13fd70 0x7f84bc139ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.461 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84baffd700 1 -- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84bc0b7160 msgr2=0x7f84bc13b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.461 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84baffd700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84bc0b7160 0x7f84bc13b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.461 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.455+0000 7f84baffd700 1 -- 172.21.15.94:0/3314351850 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00094f0 con 0x7f84bc070c50 2023-09-30T00:18:05.461 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.456+0000 7f84baffd700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc070c50 0x7f84bc0a3c40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00ce00 tx=0x7f84ac001b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.461 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.456+0000 7f84a2ffd700 1 -- 172.21.15.94:0/3314351850 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac005df0 con 0x7f84bc070c50 2023-09-30T00:18:05.461 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.456+0000 7f84a2ffd700 1 -- 172.21.15.94:0/3314351850 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac00a9b0 con 0x7f84bc070c50 2023-09-30T00:18:05.461 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.456+0000 7f84a2ffd700 1 -- 172.21.15.94:0/3314351850 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac00ad70 con 0x7f84bc070c50 2023-09-30T00:18:05.461 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.456+0000 7f84c1303700 1 -- 172.21.15.94:0/3314351850 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b4009420 con 0x7f84bc070c50 2023-09-30T00:18:05.461 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.456+0000 7f84c1303700 1 -- 172.21.15.94:0/3314351850 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84bc09b790 con 0x7f84bc070c50 2023-09-30T00:18:05.462 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.457+0000 7f84c1303700 1 -- 172.21.15.94:0/3314351850 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84bc071630 con 0x7f84bc070c50 2023-09-30T00:18:05.462 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.457+0000 7f84a2ffd700 1 -- 172.21.15.94:0/3314351850 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f84ac007300 con 0x7f84bc070c50 2023-09-30T00:18:05.462 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.458+0000 7f84a2ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:05.462 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.458+0000 7f84a2ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:05.462 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.458+0000 7f84a2ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:05.462 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.458+0000 7f84a2ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:05.462 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.458+0000 7f84a2ffd700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f84a8098780 0x7f84a809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.463 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.458+0000 7f84ba7fc700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f84a8098780 0x7f84a809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.463 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.458+0000 7f84a2ffd700 1 -- 172.21.15.94:0/3314351850 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f84ac0c0580 con 0x7f84bc070c50 2023-09-30T00:18:05.463 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.459+0000 7f84ba7fc700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f84a8098780 0x7f84a809ac30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f84bc13da40 tx=0x7f84b0009250 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.463 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.462+0000 7f84a2ffd700 1 -- 172.21.15.94:0/3314351850 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f84ac08dc30 con 0x7f84bc070c50 2023-09-30T00:18:05.641 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.640+0000 7f84c1303700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-30T00:18:05.641 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.640+0000 7f84c1303700 1 -- 172.21.15.94:0/3314351850 --> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f84bc0a1310 con 0x7f84a8098780 2023-09-30T00:18:05.647 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.647+0000 7f84a2ffd700 1 -- 172.21.15.94:0/3314351850 <== mgr.4106 v2:172.21.15.94:6808/84242 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+269351 (secure 0 0 0) 0x7f84bc0a1310 con 0x7f84a8098780 2023-09-30T00:18:05.647 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.647+0000 7f84a2ffd700 20 mgrc handle_command_reply tid 0 r 0 2023-09-30T00:18:05.651 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:05.653 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 10 mgrc shutdown 2023-09-30T00:18:05.653 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 1 -- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f84a8098780 msgr2=0x7f84a809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.653 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f84a8098780 0x7f84a809ac30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f84bc13da40 tx=0x7f84b0009250 comp rx=0 tx=0).stop 2023-09-30T00:18:05.653 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 1 -- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc070c50 msgr2=0x7f84bc0a3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.653 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc070c50 0x7f84bc0a3c40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00ce00 tx=0x7f84ac001b90 comp rx=0 tx=0).stop 2023-09-30T00:18:05.654 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 1 -- 172.21.15.94:0/3314351850 shutdown_connections 2023-09-30T00:18:05.654 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f84a8098780 0x7f84a809ac30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.654 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84bc070c50 0x7f84bc0a3c40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.654 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84bc13fd70 0x7f84bc139ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.654 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 1 --2- 172.21.15.94:0/3314351850 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84bc0b7160 0x7f84bc13b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.654 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 1 -- 172.21.15.94:0/3314351850 >> 172.21.15.94:0/3314351850 conn(0x7f84bc06a090 msgr2=0x7f84bc067cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:05.654 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 1 -- 172.21.15.94:0/3314351850 shutdown_connections 2023-09-30T00:18:05.654 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.650+0000 7f84c1303700 1 -- 172.21.15.94:0/3314351850 wait complete. 2023-09-30T00:18:05.654 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2023-09-30T00:18:05.663 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-09-30T00:18:03.852953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":34452,"kb_used_data":1908,"kb_used_omap":0,"kb_used_meta":32512,"kb_avail":377452908,"statfs":{"total":386547056640,"available":386511777792,"internally_reserved":0,"allocated":1953792,"data_stored":1422368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33292288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.909897"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962397+0000","last_change":"2023-09-30T00:17:51.960193+0000","last_active":"2023-09-30T00:17:51.962397+0000","last_peered":"2023-09-30T00:17:51.962397+0000","last_clean":"2023-09-30T00:17:51.962397+0000","last_became_active":"2023-09-30T00:17:51.959888+0000","last_became_peered":"2023-09-30T00:17:51.959888+0000","last_unstale":"2023-09-30T00:17:51.962397+0000","last_undegraded":"2023-09-30T00:17:51.962397+0000","last_fullsized":"2023-09-30T00:17:51.962397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:40:42.458928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972435+0000","last_change":"2023-09-30T00:17:51.968577+0000","last_active":"2023-09-30T00:17:51.972435+0000","last_peered":"2023-09-30T00:17:51.972435+0000","last_clean":"2023-09-30T00:17:51.972435+0000","last_became_active":"2023-09-30T00:17:51.968275+0000","last_became_peered":"2023-09-30T00:17:51.968275+0000","last_unstale":"2023-09-30T00:17:51.972435+0000","last_undegraded":"2023-09-30T00:17:51.972435+0000","last_fullsized":"2023-09-30T00:17:51.972435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:19:24.974156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977703+0000","last_change":"2023-09-30T00:17:51.974359+0000","last_active":"2023-09-30T00:17:51.977703+0000","last_peered":"2023-09-30T00:17:51.977703+0000","last_clean":"2023-09-30T00:17:51.977703+0000","last_became_active":"2023-09-30T00:17:51.973789+0000","last_became_peered":"2023-09-30T00:17:51.973789+0000","last_unstale":"2023-09-30T00:17:51.977703+0000","last_undegraded":"2023-09-30T00:17:51.977703+0000","last_fullsized":"2023-09-30T00:17:51.977703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:27.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971859+0000","last_change":"2023-09-30T00:17:51.958127+0000","last_active":"2023-09-30T00:17:51.971859+0000","last_peered":"2023-09-30T00:17:51.971859+0000","last_clean":"2023-09-30T00:17:51.971859+0000","last_became_active":"2023-09-30T00:17:51.957850+0000","last_became_peered":"2023-09-30T00:17:51.957850+0000","last_unstale":"2023-09-30T00:17:51.971859+0000","last_undegraded":"2023-09-30T00:17:51.971859+0000","last_fullsized":"2023-09-30T00:17:51.971859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:48:36.950659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973474+0000","last_change":"2023-09-30T00:17:51.966792+0000","last_active":"2023-09-30T00:17:51.973474+0000","last_peered":"2023-09-30T00:17:51.973474+0000","last_clean":"2023-09-30T00:17:51.973474+0000","last_became_active":"2023-09-30T00:17:51.965887+0000","last_became_peered":"2023-09-30T00:17:51.965887+0000","last_unstale":"2023-09-30T00:17:51.973474+0000","last_undegraded":"2023-09-30T00:17:51.973474+0000","last_fullsized":"2023-09-30T00:17:51.973474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973636+0000","last_change":"2023-09-30T00:17:51.967175+0000","last_active":"2023-09-30T00:17:51.973636+0000","last_peered":"2023-09-30T00:17:51.973636+0000","last_clean":"2023-09-30T00:17:51.973636+0000","last_became_active":"2023-09-30T00:17:51.966408+0000","last_became_peered":"2023-09-30T00:17:51.966408+0000","last_unstale":"2023-09-30T00:17:51.973636+0000","last_undegraded":"2023-09-30T00:17:51.973636+0000","last_fullsized":"2023-09-30T00:17:51.973636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:47:56.029102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972783+0000","last_change":"2023-09-30T00:17:51.964264+0000","last_active":"2023-09-30T00:17:51.972783+0000","last_peered":"2023-09-30T00:17:51.972783+0000","last_clean":"2023-09-30T00:17:51.972783+0000","last_became_active":"2023-09-30T00:17:51.962740+0000","last_became_peered":"2023-09-30T00:17:51.962740+0000","last_unstale":"2023-09-30T00:17:51.972783+0000","last_undegraded":"2023-09-30T00:17:51.972783+0000","last_fullsized":"2023-09-30T00:17:51.972783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:52:26.654463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975296+0000","last_change":"2023-09-30T00:17:51.967599+0000","last_active":"2023-09-30T00:17:51.975296+0000","last_peered":"2023-09-30T00:17:51.975296+0000","last_clean":"2023-09-30T00:17:51.975296+0000","last_became_active":"2023-09-30T00:17:51.961762+0000","last_became_peered":"2023-09-30T00:17:51.961762+0000","last_unstale":"2023-09-30T00:17:51.975296+0000","last_undegraded":"2023-09-30T00:17:51.975296+0000","last_fullsized":"2023-09-30T00:17:51.975296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:35:31.620523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971780+0000","last_change":"2023-09-30T00:17:51.971131+0000","last_active":"2023-09-30T00:17:51.971780+0000","last_peered":"2023-09-30T00:17:51.971780+0000","last_clean":"2023-09-30T00:17:51.971780+0000","last_became_active":"2023-09-30T00:17:51.970865+0000","last_became_peered":"2023-09-30T00:17:51.970865+0000","last_unstale":"2023-09-30T00:17:51.971780+0000","last_undegraded":"2023-09-30T00:17:51.971780+0000","last_fullsized":"2023-09-30T00:17:51.971780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:53:33.380570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972400+0000","last_change":"2023-09-30T00:17:51.963624+0000","last_active":"2023-09-30T00:17:51.972400+0000","last_peered":"2023-09-30T00:17:51.972400+0000","last_clean":"2023-09-30T00:17:51.972400+0000","last_became_active":"2023-09-30T00:17:51.963303+0000","last_became_peered":"2023-09-30T00:17:51.963303+0000","last_unstale":"2023-09-30T00:17:51.972400+0000","last_undegraded":"2023-09-30T00:17:51.972400+0000","last_fullsized":"2023-09-30T00:17:51.972400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971954+0000","last_change":"2023-09-30T00:17:51.961620+0000","last_active":"2023-09-30T00:17:51.971954+0000","last_peered":"2023-09-30T00:17:51.971954+0000","last_clean":"2023-09-30T00:17:51.971954+0000","last_became_active":"2023-09-30T00:17:51.958237+0000","last_became_peered":"2023-09-30T00:17:51.958237+0000","last_unstale":"2023-09-30T00:17:51.971954+0000","last_undegraded":"2023-09-30T00:17:51.971954+0000","last_fullsized":"2023-09-30T00:17:51.971954+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977575+0000","last_change":"2023-09-30T00:17:51.974397+0000","last_active":"2023-09-30T00:17:51.977575+0000","last_peered":"2023-09-30T00:17:51.977575+0000","last_clean":"2023-09-30T00:17:51.977575+0000","last_became_active":"2023-09-30T00:17:51.974025+0000","last_became_peered":"2023-09-30T00:17:51.974025+0000","last_unstale":"2023-09-30T00:17:51.977575+0000","last_undegraded":"2023-09-30T00:17:51.977575+0000","last_fullsized":"2023-09-30T00:17:51.977575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:49:07.723751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.966627+0000","last_change":"2023-09-30T00:17:51.966187+0000","last_active":"2023-09-30T00:17:51.966627+0000","last_peered":"2023-09-30T00:17:51.966627+0000","last_clean":"2023-09-30T00:17:51.966627+0000","last_became_active":"2023-09-30T00:17:51.965900+0000","last_became_peered":"2023-09-30T00:17:51.965900+0000","last_unstale":"2023-09-30T00:17:51.966627+0000","last_undegraded":"2023-09-30T00:17:51.966627+0000","last_fullsized":"2023-09-30T00:17:51.966627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:30:07.208946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976147+0000","last_change":"2023-09-30T00:17:51.970985+0000","last_active":"2023-09-30T00:17:51.976147+0000","last_peered":"2023-09-30T00:17:51.976147+0000","last_clean":"2023-09-30T00:17:51.976147+0000","last_became_active":"2023-09-30T00:17:51.966979+0000","last_became_peered":"2023-09-30T00:17:51.966979+0000","last_unstale":"2023-09-30T00:17:51.976147+0000","last_undegraded":"2023-09-30T00:17:51.976147+0000","last_fullsized":"2023-09-30T00:17:51.976147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:09:28.833739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975911+0000","last_change":"2023-09-30T00:17:51.970398+0000","last_active":"2023-09-30T00:17:51.975911+0000","last_peered":"2023-09-30T00:17:51.975911+0000","last_clean":"2023-09-30T00:17:51.975911+0000","last_became_active":"2023-09-30T00:17:51.961675+0000","last_became_peered":"2023-09-30T00:17:51.961675+0000","last_unstale":"2023-09-30T00:17:51.975911+0000","last_undegraded":"2023-09-30T00:17:51.975911+0000","last_fullsized":"2023-09-30T00:17:51.975911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:33:21.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972718+0000","last_change":"2023-09-30T00:17:51.969081+0000","last_active":"2023-09-30T00:17:51.972718+0000","last_peered":"2023-09-30T00:17:51.972718+0000","last_clean":"2023-09-30T00:17:51.972718+0000","last_became_active":"2023-09-30T00:17:51.966447+0000","last_became_peered":"2023-09-30T00:17:51.966447+0000","last_unstale":"2023-09-30T00:17:51.972718+0000","last_undegraded":"2023-09-30T00:17:51.972718+0000","last_fullsized":"2023-09-30T00:17:51.972718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:19:12.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960350+0000","last_change":"2023-09-30T00:17:51.953980+0000","last_active":"2023-09-30T00:17:51.960350+0000","last_peered":"2023-09-30T00:17:51.960350+0000","last_clean":"2023-09-30T00:17:51.960350+0000","last_became_active":"2023-09-30T00:17:51.952191+0000","last_became_peered":"2023-09-30T00:17:51.952191+0000","last_unstale":"2023-09-30T00:17:51.960350+0000","last_undegraded":"2023-09-30T00:17:51.960350+0000","last_fullsized":"2023-09-30T00:17:51.960350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:52:46.774408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973554+0000","last_change":"2023-09-30T00:17:51.969589+0000","last_active":"2023-09-30T00:17:51.973554+0000","last_peered":"2023-09-30T00:17:51.973554+0000","last_clean":"2023-09-30T00:17:51.973554+0000","last_became_active":"2023-09-30T00:17:51.967300+0000","last_became_peered":"2023-09-30T00:17:51.967300+0000","last_unstale":"2023-09-30T00:17:51.973554+0000","last_undegraded":"2023-09-30T00:17:51.973554+0000","last_fullsized":"2023-09-30T00:17:51.973554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:22:57.623135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.966670+0000","last_change":"2023-09-30T00:17:51.966047+0000","last_active":"2023-09-30T00:17:51.966670+0000","last_peered":"2023-09-30T00:17:51.966670+0000","last_clean":"2023-09-30T00:17:51.966670+0000","last_became_active":"2023-09-30T00:17:51.965457+0000","last_became_peered":"2023-09-30T00:17:51.965457+0000","last_unstale":"2023-09-30T00:17:51.966670+0000","last_undegraded":"2023-09-30T00:17:51.966670+0000","last_fullsized":"2023-09-30T00:17:51.966670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:24:33.105806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972047+0000","last_change":"2023-09-30T00:17:51.961502+0000","last_active":"2023-09-30T00:17:51.972047+0000","last_peered":"2023-09-30T00:17:51.972047+0000","last_clean":"2023-09-30T00:17:51.972047+0000","last_became_active":"2023-09-30T00:17:51.961191+0000","last_became_peered":"2023-09-30T00:17:51.961191+0000","last_unstale":"2023-09-30T00:17:51.972047+0000","last_undegraded":"2023-09-30T00:17:51.972047+0000","last_fullsized":"2023-09-30T00:17:51.972047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:05:37.196886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976121+0000","last_change":"2023-09-30T00:17:51.970789+0000","last_active":"2023-09-30T00:17:51.976121+0000","last_peered":"2023-09-30T00:17:51.976121+0000","last_clean":"2023-09-30T00:17:51.976121+0000","last_became_active":"2023-09-30T00:17:51.967616+0000","last_became_peered":"2023-09-30T00:17:51.967616+0000","last_unstale":"2023-09-30T00:17:51.976121+0000","last_undegraded":"2023-09-30T00:17:51.976121+0000","last_fullsized":"2023-09-30T00:17:51.976121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976127+0000","last_change":"2023-09-30T00:17:51.970797+0000","last_active":"2023-09-30T00:17:51.976127+0000","last_peered":"2023-09-30T00:17:51.976127+0000","last_clean":"2023-09-30T00:17:51.976127+0000","last_became_active":"2023-09-30T00:17:51.963382+0000","last_became_peered":"2023-09-30T00:17:51.963382+0000","last_unstale":"2023-09-30T00:17:51.976127+0000","last_undegraded":"2023-09-30T00:17:51.976127+0000","last_fullsized":"2023-09-30T00:17:51.976127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:35:22.077360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972764+0000","last_change":"2023-09-30T00:17:51.963058+0000","last_active":"2023-09-30T00:17:51.972764+0000","last_peered":"2023-09-30T00:17:51.972764+0000","last_clean":"2023-09-30T00:17:51.972764+0000","last_became_active":"2023-09-30T00:17:51.959450+0000","last_became_peered":"2023-09-30T00:17:51.959450+0000","last_unstale":"2023-09-30T00:17:51.972764+0000","last_undegraded":"2023-09-30T00:17:51.972764+0000","last_fullsized":"2023-09-30T00:17:51.972764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:41:36.585190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975442+0000","last_change":"2023-09-30T00:17:51.967581+0000","last_active":"2023-09-30T00:17:51.975442+0000","last_peered":"2023-09-30T00:17:51.975442+0000","last_clean":"2023-09-30T00:17:51.975442+0000","last_became_active":"2023-09-30T00:17:51.961348+0000","last_became_peered":"2023-09-30T00:17:51.961348+0000","last_unstale":"2023-09-30T00:17:51.975442+0000","last_undegraded":"2023-09-30T00:17:51.975442+0000","last_fullsized":"2023-09-30T00:17:51.975442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:11:09.140469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975871+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.975871+0000","last_peered":"2023-09-30T00:17:51.975871+0000","last_clean":"2023-09-30T00:17:51.975871+0000","last_became_active":"2023-09-30T00:17:51.963534+0000","last_became_peered":"2023-09-30T00:17:51.963534+0000","last_unstale":"2023-09-30T00:17:51.975871+0000","last_undegraded":"2023-09-30T00:17:51.975871+0000","last_fullsized":"2023-09-30T00:17:51.975871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:21:49.086269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.961574+0000","last_change":"2023-09-30T00:17:51.961210+0000","last_active":"2023-09-30T00:17:51.961574+0000","last_peered":"2023-09-30T00:17:51.961574+0000","last_clean":"2023-09-30T00:17:51.961574+0000","last_became_active":"2023-09-30T00:17:51.961010+0000","last_became_peered":"2023-09-30T00:17:51.961010+0000","last_unstale":"2023-09-30T00:17:51.961574+0000","last_undegraded":"2023-09-30T00:17:51.961574+0000","last_fullsized":"2023-09-30T00:17:51.961574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:31:50.854220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972810+0000","last_change":"2023-09-30T00:17:51.968803+0000","last_active":"2023-09-30T00:17:51.972810+0000","last_peered":"2023-09-30T00:17:51.972810+0000","last_clean":"2023-09-30T00:17:51.972810+0000","last_became_active":"2023-09-30T00:17:51.965833+0000","last_became_peered":"2023-09-30T00:17:51.965833+0000","last_unstale":"2023-09-30T00:17:51.972810+0000","last_undegraded":"2023-09-30T00:17:51.972810+0000","last_fullsized":"2023-09-30T00:17:51.972810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960606+0000","last_change":"2023-09-30T00:17:51.592497+0000","last_active":"2023-09-30T00:17:51.960606+0000","last_peered":"2023-09-30T00:17:51.960606+0000","last_clean":"2023-09-30T00:17:51.960606+0000","last_became_active":"2023-09-30T00:17:51.585128+0000","last_became_peered":"2023-09-30T00:17:51.585128+0000","last_unstale":"2023-09-30T00:17:51.960606+0000","last_undegraded":"2023-09-30T00:17:51.960606+0000","last_fullsized":"2023-09-30T00:17:51.960606+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:46.058014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977744+0000","last_change":"2023-09-30T00:17:51.974520+0000","last_active":"2023-09-30T00:17:51.977744+0000","last_peered":"2023-09-30T00:17:51.977744+0000","last_clean":"2023-09-30T00:17:51.977744+0000","last_became_active":"2023-09-30T00:17:51.974109+0000","last_became_peered":"2023-09-30T00:17:51.974109+0000","last_unstale":"2023-09-30T00:17:51.977744+0000","last_undegraded":"2023-09-30T00:17:51.977744+0000","last_fullsized":"2023-09-30T00:17:51.977744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:25:05.134336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941011+0000","last_change":"2023-09-30T00:17:51.587986+0000","last_active":"2023-09-30T00:17:51.941011+0000","last_peered":"2023-09-30T00:17:51.941011+0000","last_clean":"2023-09-30T00:17:51.941011+0000","last_became_active":"2023-09-30T00:17:51.587724+0000","last_became_peered":"2023-09-30T00:17:51.587724+0000","last_unstale":"2023-09-30T00:17:51.941011+0000","last_undegraded":"2023-09-30T00:17:51.941011+0000","last_fullsized":"2023-09-30T00:17:51.941011+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971773+0000","last_change":"2023-09-30T00:17:51.971105+0000","last_active":"2023-09-30T00:17:51.971773+0000","last_peered":"2023-09-30T00:17:51.971773+0000","last_clean":"2023-09-30T00:17:51.971773+0000","last_became_active":"2023-09-30T00:17:51.970846+0000","last_became_peered":"2023-09-30T00:17:51.970846+0000","last_unstale":"2023-09-30T00:17:51.971773+0000","last_undegraded":"2023-09-30T00:17:51.971773+0000","last_fullsized":"2023-09-30T00:17:51.971773+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:51.108522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952700+0000","last_change":"2023-09-30T00:17:51.596511+0000","last_active":"2023-09-30T00:17:51.952700+0000","last_peered":"2023-09-30T00:17:51.952700+0000","last_clean":"2023-09-30T00:17:51.952700+0000","last_became_active":"2023-09-30T00:17:51.596034+0000","last_became_peered":"2023-09-30T00:17:51.596034+0000","last_unstale":"2023-09-30T00:17:51.952700+0000","last_undegraded":"2023-09-30T00:17:51.952700+0000","last_fullsized":"2023-09-30T00:17:51.952700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:44:29.988610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969658+0000","last_change":"2023-09-30T00:17:51.964766+0000","last_active":"2023-09-30T00:17:51.969658+0000","last_peered":"2023-09-30T00:17:51.969658+0000","last_clean":"2023-09-30T00:17:51.969658+0000","last_became_active":"2023-09-30T00:17:51.964496+0000","last_became_peered":"2023-09-30T00:17:51.964496+0000","last_unstale":"2023-09-30T00:17:51.969658+0000","last_undegraded":"2023-09-30T00:17:51.969658+0000","last_fullsized":"2023-09-30T00:17:51.969658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.942081+0000","last_change":"2023-09-30T00:17:51.577022+0000","last_active":"2023-09-30T00:17:51.942081+0000","last_peered":"2023-09-30T00:17:51.942081+0000","last_clean":"2023-09-30T00:17:51.942081+0000","last_became_active":"2023-09-30T00:17:51.576726+0000","last_became_peered":"2023-09-30T00:17:51.576726+0000","last_unstale":"2023-09-30T00:17:51.942081+0000","last_undegraded":"2023-09-30T00:17:51.942081+0000","last_fullsized":"2023-09-30T00:17:51.942081+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972739+0000","last_change":"2023-09-30T00:17:51.969564+0000","last_active":"2023-09-30T00:17:51.972739+0000","last_peered":"2023-09-30T00:17:51.972739+0000","last_clean":"2023-09-30T00:17:51.972739+0000","last_became_active":"2023-09-30T00:17:51.969240+0000","last_became_peered":"2023-09-30T00:17:51.969240+0000","last_unstale":"2023-09-30T00:17:51.972739+0000","last_undegraded":"2023-09-30T00:17:51.972739+0000","last_fullsized":"2023-09-30T00:17:51.972739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:42.701890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.949738+0000","last_change":"2023-09-30T00:17:51.588716+0000","last_active":"2023-09-30T00:17:51.949738+0000","last_peered":"2023-09-30T00:17:51.949738+0000","last_clean":"2023-09-30T00:17:51.949738+0000","last_became_active":"2023-09-30T00:17:51.588454+0000","last_became_peered":"2023-09-30T00:17:51.588454+0000","last_unstale":"2023-09-30T00:17:51.949738+0000","last_undegraded":"2023-09-30T00:17:51.949738+0000","last_fullsized":"2023-09-30T00:17:51.949738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.958204+0000","last_change":"2023-09-30T00:17:51.957598+0000","last_active":"2023-09-30T00:17:51.958204+0000","last_peered":"2023-09-30T00:17:51.958204+0000","last_clean":"2023-09-30T00:17:51.958204+0000","last_became_active":"2023-09-30T00:17:51.957336+0000","last_became_peered":"2023-09-30T00:17:51.957336+0000","last_unstale":"2023-09-30T00:17:51.958204+0000","last_undegraded":"2023-09-30T00:17:51.958204+0000","last_fullsized":"2023-09-30T00:17:51.958204+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:02:32.124291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"18'3","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973987+0000","last_change":"2023-09-30T00:17:51.589988+0000","last_active":"2023-09-30T00:17:51.973987+0000","last_peered":"2023-09-30T00:17:51.973987+0000","last_clean":"2023-09-30T00:17:51.973987+0000","last_became_active":"2023-09-30T00:17:51.589729+0000","last_became_peered":"2023-09-30T00:17:51.589729+0000","last_unstale":"2023-09-30T00:17:51.973987+0000","last_undegraded":"2023-09-30T00:17:51.973987+0000","last_fullsized":"2023-09-30T00:17:51.973987+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975239+0000","last_change":"2023-09-30T00:17:51.967245+0000","last_active":"2023-09-30T00:17:51.975239+0000","last_peered":"2023-09-30T00:17:51.975239+0000","last_clean":"2023-09-30T00:17:51.975239+0000","last_became_active":"2023-09-30T00:17:51.963755+0000","last_became_peered":"2023-09-30T00:17:51.963755+0000","last_unstale":"2023-09-30T00:17:51.975239+0000","last_undegraded":"2023-09-30T00:17:51.975239+0000","last_fullsized":"2023-09-30T00:17:51.975239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:57:31.735059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"18'6","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.983073+0000","last_change":"2023-09-30T00:17:51.592307+0000","last_active":"2023-09-30T00:17:51.983073+0000","last_peered":"2023-09-30T00:17:51.983073+0000","last_clean":"2023-09-30T00:17:51.983073+0000","last_became_active":"2023-09-30T00:17:51.591753+0000","last_became_peered":"2023-09-30T00:17:51.591753+0000","last_unstale":"2023-09-30T00:17:51.983073+0000","last_undegraded":"2023-09-30T00:17:51.983073+0000","last_fullsized":"2023-09-30T00:17:51.983073+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:14:46.193926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968817+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.968817+0000","last_peered":"2023-09-30T00:17:51.968817+0000","last_clean":"2023-09-30T00:17:51.968817+0000","last_became_active":"2023-09-30T00:17:51.967118+0000","last_became_peered":"2023-09-30T00:17:51.967118+0000","last_unstale":"2023-09-30T00:17:51.968817+0000","last_undegraded":"2023-09-30T00:17:51.968817+0000","last_fullsized":"2023-09-30T00:17:51.968817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:50.694814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969722+0000","last_change":"2023-09-30T00:17:51.591516+0000","last_active":"2023-09-30T00:17:51.969722+0000","last_peered":"2023-09-30T00:17:51.969722+0000","last_clean":"2023-09-30T00:17:51.969722+0000","last_became_active":"2023-09-30T00:17:51.591206+0000","last_became_peered":"2023-09-30T00:17:51.591206+0000","last_unstale":"2023-09-30T00:17:51.969722+0000","last_undegraded":"2023-09-30T00:17:51.969722+0000","last_fullsized":"2023-09-30T00:17:51.969722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:22.807978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977754+0000","last_change":"2023-09-30T00:17:51.974628+0000","last_active":"2023-09-30T00:17:51.977754+0000","last_peered":"2023-09-30T00:17:51.977754+0000","last_clean":"2023-09-30T00:17:51.977754+0000","last_became_active":"2023-09-30T00:17:51.974276+0000","last_became_peered":"2023-09-30T00:17:51.974276+0000","last_unstale":"2023-09-30T00:17:51.977754+0000","last_undegraded":"2023-09-30T00:17:51.977754+0000","last_fullsized":"2023-09-30T00:17:51.977754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.946238+0000","last_change":"2023-09-30T00:17:51.589138+0000","last_active":"2023-09-30T00:17:51.946238+0000","last_peered":"2023-09-30T00:17:51.946238+0000","last_clean":"2023-09-30T00:17:51.946238+0000","last_became_active":"2023-09-30T00:17:51.588883+0000","last_became_peered":"2023-09-30T00:17:51.588883+0000","last_unstale":"2023-09-30T00:17:51.946238+0000","last_undegraded":"2023-09-30T00:17:51.946238+0000","last_fullsized":"2023-09-30T00:17:51.946238+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977625+0000","last_change":"2023-09-30T00:17:51.972690+0000","last_active":"2023-09-30T00:17:51.977625+0000","last_peered":"2023-09-30T00:17:51.977625+0000","last_clean":"2023-09-30T00:17:51.977625+0000","last_became_active":"2023-09-30T00:17:51.969872+0000","last_became_peered":"2023-09-30T00:17:51.969872+0000","last_unstale":"2023-09-30T00:17:51.977625+0000","last_undegraded":"2023-09-30T00:17:51.977625+0000","last_fullsized":"2023-09-30T00:17:51.977625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952325+0000","last_change":"2023-09-30T00:17:51.597410+0000","last_active":"2023-09-30T00:17:51.952325+0000","last_peered":"2023-09-30T00:17:51.952325+0000","last_clean":"2023-09-30T00:17:51.952325+0000","last_became_active":"2023-09-30T00:17:51.597168+0000","last_became_peered":"2023-09-30T00:17:51.597168+0000","last_unstale":"2023-09-30T00:17:51.952325+0000","last_undegraded":"2023-09-30T00:17:51.952325+0000","last_fullsized":"2023-09-30T00:17:51.952325+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:44:45.339994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973574+0000","last_change":"2023-09-30T00:17:51.969752+0000","last_active":"2023-09-30T00:17:51.973574+0000","last_peered":"2023-09-30T00:17:51.973574+0000","last_clean":"2023-09-30T00:17:51.973574+0000","last_became_active":"2023-09-30T00:17:51.968993+0000","last_became_peered":"2023-09-30T00:17:51.968993+0000","last_unstale":"2023-09-30T00:17:51.973574+0000","last_undegraded":"2023-09-30T00:17:51.973574+0000","last_fullsized":"2023-09-30T00:17:51.973574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"18'8","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974927+0000","last_change":"2023-09-30T00:17:51.592339+0000","last_active":"2023-09-30T00:17:51.974927+0000","last_peered":"2023-09-30T00:17:51.974927+0000","last_clean":"2023-09-30T00:17:51.974927+0000","last_became_active":"2023-09-30T00:17:51.592134+0000","last_became_peered":"2023-09-30T00:17:51.592134+0000","last_unstale":"2023-09-30T00:17:51.974927+0000","last_undegraded":"2023-09-30T00:17:51.974927+0000","last_fullsized":"2023-09-30T00:17:51.974927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.959394+0000","last_change":"2023-09-30T00:17:51.958723+0000","last_active":"2023-09-30T00:17:51.959394+0000","last_peered":"2023-09-30T00:17:51.959394+0000","last_clean":"2023-09-30T00:17:51.959394+0000","last_became_active":"2023-09-30T00:17:51.958465+0000","last_became_peered":"2023-09-30T00:17:51.958465+0000","last_unstale":"2023-09-30T00:17:51.959394+0000","last_undegraded":"2023-09-30T00:17:51.959394+0000","last_fullsized":"2023-09-30T00:17:51.959394+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:59:59.352777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968301+0000","last_change":"2023-09-30T00:17:51.577740+0000","last_active":"2023-09-30T00:17:51.968301+0000","last_peered":"2023-09-30T00:17:51.968301+0000","last_clean":"2023-09-30T00:17:51.968301+0000","last_became_active":"2023-09-30T00:17:51.577449+0000","last_became_peered":"2023-09-30T00:17:51.577449+0000","last_unstale":"2023-09-30T00:17:51.968301+0000","last_undegraded":"2023-09-30T00:17:51.968301+0000","last_fullsized":"2023-09-30T00:17:51.968301+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977756+0000","last_change":"2023-09-30T00:17:51.974425+0000","last_active":"2023-09-30T00:17:51.977756+0000","last_peered":"2023-09-30T00:17:51.977756+0000","last_clean":"2023-09-30T00:17:51.977756+0000","last_became_active":"2023-09-30T00:17:51.974143+0000","last_became_peered":"2023-09-30T00:17:51.974143+0000","last_unstale":"2023-09-30T00:17:51.977756+0000","last_undegraded":"2023-09-30T00:17:51.977756+0000","last_fullsized":"2023-09-30T00:17:51.977756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:04:54.968664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972091+0000","last_change":"2023-09-30T00:17:51.578710+0000","last_active":"2023-09-30T00:17:51.972091+0000","last_peered":"2023-09-30T00:17:51.972091+0000","last_clean":"2023-09-30T00:17:51.972091+0000","last_became_active":"2023-09-30T00:17:51.578450+0000","last_became_peered":"2023-09-30T00:17:51.578450+0000","last_unstale":"2023-09-30T00:17:51.972091+0000","last_undegraded":"2023-09-30T00:17:51.972091+0000","last_fullsized":"2023-09-30T00:17:51.972091+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.958310+0000","last_change":"2023-09-30T00:17:51.951367+0000","last_active":"2023-09-30T00:17:51.958310+0000","last_peered":"2023-09-30T00:17:51.958310+0000","last_clean":"2023-09-30T00:17:51.958310+0000","last_became_active":"2023-09-30T00:17:51.951031+0000","last_became_peered":"2023-09-30T00:17:51.951031+0000","last_unstale":"2023-09-30T00:17:51.958310+0000","last_undegraded":"2023-09-30T00:17:51.958310+0000","last_fullsized":"2023-09-30T00:17:51.958310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976462+0000","last_change":"2023-09-30T00:17:51.594234+0000","last_active":"2023-09-30T00:17:51.976462+0000","last_peered":"2023-09-30T00:17:51.976462+0000","last_clean":"2023-09-30T00:17:51.976462+0000","last_became_active":"2023-09-30T00:17:51.594049+0000","last_became_peered":"2023-09-30T00:17:51.594049+0000","last_unstale":"2023-09-30T00:17:51.976462+0000","last_undegraded":"2023-09-30T00:17:51.976462+0000","last_fullsized":"2023-09-30T00:17:51.976462+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:53:48.206055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968844+0000","last_change":"2023-09-30T00:17:51.965298+0000","last_active":"2023-09-30T00:17:51.968844+0000","last_peered":"2023-09-30T00:17:51.968844+0000","last_clean":"2023-09-30T00:17:51.968844+0000","last_became_active":"2023-09-30T00:17:51.964965+0000","last_became_peered":"2023-09-30T00:17:51.964965+0000","last_unstale":"2023-09-30T00:17:51.968844+0000","last_undegraded":"2023-09-30T00:17:51.968844+0000","last_fullsized":"2023-09-30T00:17:51.968844+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:33:20.583162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.955628+0000","last_change":"2023-09-30T00:17:51.592485+0000","last_active":"2023-09-30T00:17:51.955628+0000","last_peered":"2023-09-30T00:17:51.955628+0000","last_clean":"2023-09-30T00:17:51.955628+0000","last_became_active":"2023-09-30T00:17:51.592054+0000","last_became_peered":"2023-09-30T00:17:51.592054+0000","last_unstale":"2023-09-30T00:17:51.955628+0000","last_undegraded":"2023-09-30T00:17:51.955628+0000","last_fullsized":"2023-09-30T00:17:51.955628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:46:52.872573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.961561+0000","last_change":"2023-09-30T00:17:51.952057+0000","last_active":"2023-09-30T00:17:51.961561+0000","last_peered":"2023-09-30T00:17:51.961561+0000","last_clean":"2023-09-30T00:17:51.961561+0000","last_became_active":"2023-09-30T00:17:51.951537+0000","last_became_peered":"2023-09-30T00:17:51.951537+0000","last_unstale":"2023-09-30T00:17:51.961561+0000","last_undegraded":"2023-09-30T00:17:51.961561+0000","last_fullsized":"2023-09-30T00:17:51.961561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:20:25.199612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.966041+0000","last_change":"2023-09-30T00:17:51.578071+0000","last_active":"2023-09-30T00:17:51.966041+0000","last_peered":"2023-09-30T00:17:51.966041+0000","last_clean":"2023-09-30T00:17:51.966041+0000","last_became_active":"2023-09-30T00:17:51.577808+0000","last_became_peered":"2023-09-30T00:17:51.577808+0000","last_unstale":"2023-09-30T00:17:51.966041+0000","last_undegraded":"2023-09-30T00:17:51.966041+0000","last_fullsized":"2023-09-30T00:17:51.966041+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:20.626316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976183+0000","last_change":"2023-09-30T00:17:51.970822+0000","last_active":"2023-09-30T00:17:51.976183+0000","last_peered":"2023-09-30T00:17:51.976183+0000","last_clean":"2023-09-30T00:17:51.976183+0000","last_became_active":"2023-09-30T00:17:51.970520+0000","last_became_peered":"2023-09-30T00:17:51.970520+0000","last_unstale":"2023-09-30T00:17:51.976183+0000","last_undegraded":"2023-09-30T00:17:51.976183+0000","last_fullsized":"2023-09-30T00:17:51.976183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:07.347359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952250+0000","last_change":"2023-09-30T00:17:51.595723+0000","last_active":"2023-09-30T00:17:51.952250+0000","last_peered":"2023-09-30T00:17:51.952250+0000","last_clean":"2023-09-30T00:17:51.952250+0000","last_became_active":"2023-09-30T00:17:51.595473+0000","last_became_peered":"2023-09-30T00:17:51.595473+0000","last_unstale":"2023-09-30T00:17:51.952250+0000","last_undegraded":"2023-09-30T00:17:51.952250+0000","last_fullsized":"2023-09-30T00:17:51.952250+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:06.343890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980343+0000","last_change":"2023-09-30T00:17:51.977010+0000","last_active":"2023-09-30T00:17:51.980343+0000","last_peered":"2023-09-30T00:17:51.980343+0000","last_clean":"2023-09-30T00:17:51.980343+0000","last_became_active":"2023-09-30T00:17:51.976744+0000","last_became_peered":"2023-09-30T00:17:51.976744+0000","last_unstale":"2023-09-30T00:17:51.980343+0000","last_undegraded":"2023-09-30T00:17:51.980343+0000","last_fullsized":"2023-09-30T00:17:51.980343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:02:14.797729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952109+0000","last_change":"2023-09-30T00:17:51.595305+0000","last_active":"2023-09-30T00:17:51.952109+0000","last_peered":"2023-09-30T00:17:51.952109+0000","last_clean":"2023-09-30T00:17:51.952109+0000","last_became_active":"2023-09-30T00:17:51.595042+0000","last_became_peered":"2023-09-30T00:17:51.595042+0000","last_unstale":"2023-09-30T00:17:51.952109+0000","last_undegraded":"2023-09-30T00:17:51.952109+0000","last_fullsized":"2023-09-30T00:17:51.952109+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:54.681232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968754+0000","last_change":"2023-09-30T00:17:51.965816+0000","last_active":"2023-09-30T00:17:51.968754+0000","last_peered":"2023-09-30T00:17:51.968754+0000","last_clean":"2023-09-30T00:17:51.968754+0000","last_became_active":"2023-09-30T00:17:51.965549+0000","last_became_peered":"2023-09-30T00:17:51.965549+0000","last_unstale":"2023-09-30T00:17:51.968754+0000","last_undegraded":"2023-09-30T00:17:51.968754+0000","last_fullsized":"2023-09-30T00:17:51.968754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:30.526216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.953542+0000","last_change":"2023-09-30T00:17:51.596289+0000","last_active":"2023-09-30T00:17:51.953542+0000","last_peered":"2023-09-30T00:17:51.953542+0000","last_clean":"2023-09-30T00:17:51.953542+0000","last_became_active":"2023-09-30T00:17:51.595689+0000","last_became_peered":"2023-09-30T00:17:51.595689+0000","last_unstale":"2023-09-30T00:17:51.953542+0000","last_undegraded":"2023-09-30T00:17:51.953542+0000","last_fullsized":"2023-09-30T00:17:51.953542+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:46:55.016362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975889+0000","last_change":"2023-09-30T00:17:51.970420+0000","last_active":"2023-09-30T00:17:51.975889+0000","last_peered":"2023-09-30T00:17:51.975889+0000","last_clean":"2023-09-30T00:17:51.975889+0000","last_became_active":"2023-09-30T00:17:51.963999+0000","last_became_peered":"2023-09-30T00:17:51.963999+0000","last_unstale":"2023-09-30T00:17:51.975889+0000","last_undegraded":"2023-09-30T00:17:51.975889+0000","last_fullsized":"2023-09-30T00:17:51.975889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962989+0000","last_change":"2023-09-30T00:17:51.590485+0000","last_active":"2023-09-30T00:17:51.962989+0000","last_peered":"2023-09-30T00:17:51.962989+0000","last_clean":"2023-09-30T00:17:51.962989+0000","last_became_active":"2023-09-30T00:17:51.590210+0000","last_became_peered":"2023-09-30T00:17:51.590210+0000","last_unstale":"2023-09-30T00:17:51.962989+0000","last_undegraded":"2023-09-30T00:17:51.962989+0000","last_fullsized":"2023-09-30T00:17:51.962989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:32.708671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.944019+0000","last_change":"2023-09-30T00:17:45.196688+0000","last_active":"2023-09-30T00:17:51.944019+0000","last_peered":"2023-09-30T00:17:51.944019+0000","last_clean":"2023-09-30T00:17:51.944019+0000","last_became_active":"2023-09-30T00:17:45.196361+0000","last_became_peered":"2023-09-30T00:17:45.196361+0000","last_unstale":"2023-09-30T00:17:51.944019+0000","last_undegraded":"2023-09-30T00:17:51.944019+0000","last_fullsized":"2023-09-30T00:17:51.944019+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977735+0000","last_change":"2023-09-30T00:17:51.974386+0000","last_active":"2023-09-30T00:17:51.977735+0000","last_peered":"2023-09-30T00:17:51.977735+0000","last_clean":"2023-09-30T00:17:51.977735+0000","last_became_active":"2023-09-30T00:17:51.973776+0000","last_became_peered":"2023-09-30T00:17:51.973776+0000","last_unstale":"2023-09-30T00:17:51.977735+0000","last_undegraded":"2023-09-30T00:17:51.977735+0000","last_fullsized":"2023-09-30T00:17:51.977735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:25:21.282436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943386+0000","last_change":"2023-09-30T00:17:45.203067+0000","last_active":"2023-09-30T00:17:51.943386+0000","last_peered":"2023-09-30T00:17:51.943386+0000","last_clean":"2023-09-30T00:17:51.943386+0000","last_became_active":"2023-09-30T00:17:45.202823+0000","last_became_peered":"2023-09-30T00:17:45.202823+0000","last_unstale":"2023-09-30T00:17:51.943386+0000","last_undegraded":"2023-09-30T00:17:51.943386+0000","last_fullsized":"2023-09-30T00:17:51.943386+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:26:01.195302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-09-30T00:17:52.947337+0000","last_change":"2023-09-30T00:17:51.592843+0000","last_active":"2023-09-30T00:17:52.947337+0000","last_peered":"2023-09-30T00:17:52.947337+0000","last_clean":"2023-09-30T00:17:52.947337+0000","last_became_active":"2023-09-30T00:17:51.592601+0000","last_became_peered":"2023-09-30T00:17:51.592601+0000","last_unstale":"2023-09-30T00:17:52.947337+0000","last_undegraded":"2023-09-30T00:17:52.947337+0000","last_fullsized":"2023-09-30T00:17:52.947337+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968789+0000","last_change":"2023-09-30T00:17:51.965286+0000","last_active":"2023-09-30T00:17:51.968789+0000","last_peered":"2023-09-30T00:17:51.968789+0000","last_clean":"2023-09-30T00:17:51.968789+0000","last_became_active":"2023-09-30T00:17:51.964869+0000","last_became_peered":"2023-09-30T00:17:51.964869+0000","last_unstale":"2023-09-30T00:17:51.968789+0000","last_undegraded":"2023-09-30T00:17:51.968789+0000","last_fullsized":"2023-09-30T00:17:51.968789+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:46.192135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943060+0000","last_change":"2023-09-30T00:17:45.195737+0000","last_active":"2023-09-30T00:17:51.943060+0000","last_peered":"2023-09-30T00:17:51.943060+0000","last_clean":"2023-09-30T00:17:51.943060+0000","last_became_active":"2023-09-30T00:17:45.195464+0000","last_became_peered":"2023-09-30T00:17:45.195464+0000","last_unstale":"2023-09-30T00:17:51.943060+0000","last_undegraded":"2023-09-30T00:17:51.943060+0000","last_fullsized":"2023-09-30T00:17:51.943060+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:24:48.730978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952759+0000","last_change":"2023-09-30T00:17:47.199132+0000","last_active":"2023-09-30T00:17:51.952759+0000","last_peered":"2023-09-30T00:17:51.952759+0000","last_clean":"2023-09-30T00:17:51.952759+0000","last_became_active":"2023-09-30T00:17:47.198255+0000","last_became_peered":"2023-09-30T00:17:47.198255+0000","last_unstale":"2023-09-30T00:17:51.952759+0000","last_undegraded":"2023-09-30T00:17:51.952759+0000","last_fullsized":"2023-09-30T00:17:51.952759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_clean_scrub_stamp":"2023-09-30T00:17:46.189460+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:09.834201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943061+0000","last_change":"2023-09-30T00:17:51.575046+0000","last_active":"2023-09-30T00:17:51.943061+0000","last_peered":"2023-09-30T00:17:51.943061+0000","last_clean":"2023-09-30T00:17:51.943061+0000","last_became_active":"2023-09-30T00:17:51.574815+0000","last_became_peered":"2023-09-30T00:17:51.574815+0000","last_unstale":"2023-09-30T00:17:51.943061+0000","last_undegraded":"2023-09-30T00:17:51.943061+0000","last_fullsized":"2023-09-30T00:17:51.943061+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:43:57.892326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973520+0000","last_change":"2023-09-30T00:17:51.969276+0000","last_active":"2023-09-30T00:17:51.973520+0000","last_peered":"2023-09-30T00:17:51.973520+0000","last_clean":"2023-09-30T00:17:51.973520+0000","last_became_active":"2023-09-30T00:17:51.966513+0000","last_became_peered":"2023-09-30T00:17:51.966513+0000","last_unstale":"2023-09-30T00:17:51.973520+0000","last_undegraded":"2023-09-30T00:17:51.973520+0000","last_fullsized":"2023-09-30T00:17:51.973520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.946321+0000","last_change":"2023-09-30T00:17:45.197230+0000","last_active":"2023-09-30T00:17:51.946321+0000","last_peered":"2023-09-30T00:17:51.946321+0000","last_clean":"2023-09-30T00:17:51.946321+0000","last_became_active":"2023-09-30T00:17:45.196528+0000","last_became_peered":"2023-09-30T00:17:45.196528+0000","last_unstale":"2023-09-30T00:17:51.946321+0000","last_undegraded":"2023-09-30T00:17:51.946321+0000","last_fullsized":"2023-09-30T00:17:51.946321+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941919+0000","last_change":"2023-09-30T00:17:51.575520+0000","last_active":"2023-09-30T00:17:51.941919+0000","last_peered":"2023-09-30T00:17:51.941919+0000","last_clean":"2023-09-30T00:17:51.941919+0000","last_became_active":"2023-09-30T00:17:51.575306+0000","last_became_peered":"2023-09-30T00:17:51.575306+0000","last_unstale":"2023-09-30T00:17:51.941919+0000","last_undegraded":"2023-09-30T00:17:51.941919+0000","last_fullsized":"2023-09-30T00:17:51.941919+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:31:32.127076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.959411+0000","last_change":"2023-09-30T00:17:51.958852+0000","last_active":"2023-09-30T00:17:51.959411+0000","last_peered":"2023-09-30T00:17:51.959411+0000","last_clean":"2023-09-30T00:17:51.959411+0000","last_became_active":"2023-09-30T00:17:51.958570+0000","last_became_peered":"2023-09-30T00:17:51.958570+0000","last_unstale":"2023-09-30T00:17:51.959411+0000","last_undegraded":"2023-09-30T00:17:51.959411+0000","last_fullsized":"2023-09-30T00:17:51.959411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:34:27.611487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.940896+0000","last_change":"2023-09-30T00:17:45.196542+0000","last_active":"2023-09-30T00:17:51.940896+0000","last_peered":"2023-09-30T00:17:51.940896+0000","last_clean":"2023-09-30T00:17:51.940896+0000","last_became_active":"2023-09-30T00:17:45.196123+0000","last_became_peered":"2023-09-30T00:17:45.196123+0000","last_unstale":"2023-09-30T00:17:51.940896+0000","last_undegraded":"2023-09-30T00:17:51.940896+0000","last_fullsized":"2023-09-30T00:17:51.940896+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952371+0000","last_change":"2023-09-30T00:17:51.591518+0000","last_active":"2023-09-30T00:17:51.952371+0000","last_peered":"2023-09-30T00:17:51.952371+0000","last_clean":"2023-09-30T00:17:51.952371+0000","last_became_active":"2023-09-30T00:17:51.591253+0000","last_became_peered":"2023-09-30T00:17:51.591253+0000","last_unstale":"2023-09-30T00:17:51.952371+0000","last_undegraded":"2023-09-30T00:17:51.952371+0000","last_fullsized":"2023-09-30T00:17:51.952371+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:52:46.330854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980323+0000","last_change":"2023-09-30T00:17:51.976800+0000","last_active":"2023-09-30T00:17:51.980323+0000","last_peered":"2023-09-30T00:17:51.980323+0000","last_clean":"2023-09-30T00:17:51.980323+0000","last_became_active":"2023-09-30T00:17:51.976602+0000","last_became_peered":"2023-09-30T00:17:51.976602+0000","last_unstale":"2023-09-30T00:17:51.980323+0000","last_undegraded":"2023-09-30T00:17:51.980323+0000","last_fullsized":"2023-09-30T00:17:51.980323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:53:26.550299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952216+0000","last_change":"2023-09-30T00:17:45.207123+0000","last_active":"2023-09-30T00:17:51.952216+0000","last_peered":"2023-09-30T00:17:51.952216+0000","last_clean":"2023-09-30T00:17:51.952216+0000","last_became_active":"2023-09-30T00:17:45.206750+0000","last_became_peered":"2023-09-30T00:17:45.206750+0000","last_unstale":"2023-09-30T00:17:51.952216+0000","last_undegraded":"2023-09-30T00:17:51.952216+0000","last_fullsized":"2023-09-30T00:17:51.952216+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.945122+0000","last_change":"2023-09-30T00:17:51.585515+0000","last_active":"2023-09-30T00:17:51.945122+0000","last_peered":"2023-09-30T00:17:51.945122+0000","last_clean":"2023-09-30T00:17:51.945122+0000","last_became_active":"2023-09-30T00:17:51.585262+0000","last_became_peered":"2023-09-30T00:17:51.585262+0000","last_unstale":"2023-09-30T00:17:51.945122+0000","last_undegraded":"2023-09-30T00:17:51.945122+0000","last_fullsized":"2023-09-30T00:17:51.945122+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:04.914282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960357+0000","last_change":"2023-09-30T00:17:51.959726+0000","last_active":"2023-09-30T00:17:51.960357+0000","last_peered":"2023-09-30T00:17:51.960357+0000","last_clean":"2023-09-30T00:17:51.960357+0000","last_became_active":"2023-09-30T00:17:51.959328+0000","last_became_peered":"2023-09-30T00:17:51.959328+0000","last_unstale":"2023-09-30T00:17:51.960357+0000","last_undegraded":"2023-09-30T00:17:51.960357+0000","last_fullsized":"2023-09-30T00:17:51.960357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:37:08.356297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.954829+0000","last_change":"2023-09-30T00:17:45.206971+0000","last_active":"2023-09-30T00:17:51.954829+0000","last_peered":"2023-09-30T00:17:51.954829+0000","last_clean":"2023-09-30T00:17:51.954829+0000","last_became_active":"2023-09-30T00:17:45.206693+0000","last_became_peered":"2023-09-30T00:17:45.206693+0000","last_unstale":"2023-09-30T00:17:51.954829+0000","last_undegraded":"2023-09-30T00:17:51.954829+0000","last_fullsized":"2023-09-30T00:17:51.954829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943205+0000","last_change":"2023-09-30T00:17:51.575506+0000","last_active":"2023-09-30T00:17:51.943205+0000","last_peered":"2023-09-30T00:17:51.943205+0000","last_clean":"2023-09-30T00:17:51.943205+0000","last_became_active":"2023-09-30T00:17:51.574806+0000","last_became_peered":"2023-09-30T00:17:51.574806+0000","last_unstale":"2023-09-30T00:17:51.943205+0000","last_undegraded":"2023-09-30T00:17:51.943205+0000","last_fullsized":"2023-09-30T00:17:51.943205+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:51.816991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975408+0000","last_change":"2023-09-30T00:17:51.967785+0000","last_active":"2023-09-30T00:17:51.975408+0000","last_peered":"2023-09-30T00:17:51.975408+0000","last_clean":"2023-09-30T00:17:51.975408+0000","last_became_active":"2023-09-30T00:17:51.967419+0000","last_became_peered":"2023-09-30T00:17:51.967419+0000","last_unstale":"2023-09-30T00:17:51.975408+0000","last_undegraded":"2023-09-30T00:17:51.975408+0000","last_fullsized":"2023-09-30T00:17:51.975408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941618+0000","last_change":"2023-09-30T00:17:45.203202+0000","last_active":"2023-09-30T00:17:51.941618+0000","last_peered":"2023-09-30T00:17:51.941618+0000","last_clean":"2023-09-30T00:17:51.941618+0000","last_became_active":"2023-09-30T00:17:45.202879+0000","last_became_peered":"2023-09-30T00:17:45.202879+0000","last_unstale":"2023-09-30T00:17:51.941618+0000","last_undegraded":"2023-09-30T00:17:51.941618+0000","last_fullsized":"2023-09-30T00:17:51.941618+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:50:42.379943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.944043+0000","last_change":"2023-09-30T00:17:51.580771+0000","last_active":"2023-09-30T00:17:51.944043+0000","last_peered":"2023-09-30T00:17:51.944043+0000","last_clean":"2023-09-30T00:17:51.944043+0000","last_became_active":"2023-09-30T00:17:51.580495+0000","last_became_peered":"2023-09-30T00:17:51.580495+0000","last_unstale":"2023-09-30T00:17:51.944043+0000","last_undegraded":"2023-09-30T00:17:51.944043+0000","last_fullsized":"2023-09-30T00:17:51.944043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971832+0000","last_change":"2023-09-30T00:17:51.957745+0000","last_active":"2023-09-30T00:17:51.971832+0000","last_peered":"2023-09-30T00:17:51.971832+0000","last_clean":"2023-09-30T00:17:51.971832+0000","last_became_active":"2023-09-30T00:17:51.955331+0000","last_became_peered":"2023-09-30T00:17:51.955331+0000","last_unstale":"2023-09-30T00:17:51.971832+0000","last_undegraded":"2023-09-30T00:17:51.971832+0000","last_fullsized":"2023-09-30T00:17:51.971832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:45.393660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973877+0000","last_change":"2023-09-30T00:17:51.596021+0000","last_active":"2023-09-30T00:17:51.973877+0000","last_peered":"2023-09-30T00:17:51.973877+0000","last_clean":"2023-09-30T00:17:51.973877+0000","last_became_active":"2023-09-30T00:17:51.595745+0000","last_became_peered":"2023-09-30T00:17:51.595745+0000","last_unstale":"2023-09-30T00:17:51.973877+0000","last_undegraded":"2023-09-30T00:17:51.973877+0000","last_fullsized":"2023-09-30T00:17:51.973877+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:44.390191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960330+0000","last_change":"2023-09-30T00:17:51.959564+0000","last_active":"2023-09-30T00:17:51.960330+0000","last_peered":"2023-09-30T00:17:51.960330+0000","last_clean":"2023-09-30T00:17:51.960330+0000","last_became_active":"2023-09-30T00:17:51.959278+0000","last_became_peered":"2023-09-30T00:17:51.959278+0000","last_unstale":"2023-09-30T00:17:51.960330+0000","last_undegraded":"2023-09-30T00:17:51.960330+0000","last_fullsized":"2023-09-30T00:17:51.960330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:55.932612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974567+0000","last_change":"2023-09-30T00:17:51.587276+0000","last_active":"2023-09-30T00:17:51.974567+0000","last_peered":"2023-09-30T00:17:51.974567+0000","last_clean":"2023-09-30T00:17:51.974567+0000","last_became_active":"2023-09-30T00:17:51.586984+0000","last_became_peered":"2023-09-30T00:17:51.586984+0000","last_unstale":"2023-09-30T00:17:51.974567+0000","last_undegraded":"2023-09-30T00:17:51.974567+0000","last_fullsized":"2023-09-30T00:17:51.974567+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960802+0000","last_change":"2023-09-30T00:17:51.953787+0000","last_active":"2023-09-30T00:17:51.960802+0000","last_peered":"2023-09-30T00:17:51.960802+0000","last_clean":"2023-09-30T00:17:51.960802+0000","last_became_active":"2023-09-30T00:17:51.951805+0000","last_became_peered":"2023-09-30T00:17:51.951805+0000","last_unstale":"2023-09-30T00:17:51.960802+0000","last_undegraded":"2023-09-30T00:17:51.960802+0000","last_fullsized":"2023-09-30T00:17:51.960802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:57:04.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965610+0000","last_change":"2023-09-30T00:17:51.593225+0000","last_active":"2023-09-30T00:17:51.965610+0000","last_peered":"2023-09-30T00:17:51.965610+0000","last_clean":"2023-09-30T00:17:51.965610+0000","last_became_active":"2023-09-30T00:17:51.592913+0000","last_became_peered":"2023-09-30T00:17:51.592913+0000","last_unstale":"2023-09-30T00:17:51.965610+0000","last_undegraded":"2023-09-30T00:17:51.965610+0000","last_fullsized":"2023-09-30T00:17:51.965610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:37:56.776343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.961497+0000","last_change":"2023-09-30T00:17:51.955058+0000","last_active":"2023-09-30T00:17:51.961497+0000","last_peered":"2023-09-30T00:17:51.961497+0000","last_clean":"2023-09-30T00:17:51.961497+0000","last_became_active":"2023-09-30T00:17:51.954787+0000","last_became_peered":"2023-09-30T00:17:51.954787+0000","last_unstale":"2023-09-30T00:17:51.961497+0000","last_undegraded":"2023-09-30T00:17:51.961497+0000","last_fullsized":"2023-09-30T00:17:51.961497+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:49:30.148218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960867+0000","last_change":"2023-09-30T00:17:51.586229+0000","last_active":"2023-09-30T00:17:51.960867+0000","last_peered":"2023-09-30T00:17:51.960867+0000","last_clean":"2023-09-30T00:17:51.960867+0000","last_became_active":"2023-09-30T00:17:51.586027+0000","last_became_peered":"2023-09-30T00:17:51.586027+0000","last_unstale":"2023-09-30T00:17:51.960867+0000","last_undegraded":"2023-09-30T00:17:51.960867+0000","last_fullsized":"2023-09-30T00:17:51.960867+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:48:13.817514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962963+0000","last_change":"2023-09-30T00:17:51.959628+0000","last_active":"2023-09-30T00:17:51.962963+0000","last_peered":"2023-09-30T00:17:51.962963+0000","last_clean":"2023-09-30T00:17:51.962963+0000","last_became_active":"2023-09-30T00:17:51.958974+0000","last_became_peered":"2023-09-30T00:17:51.958974+0000","last_unstale":"2023-09-30T00:17:51.962963+0000","last_undegraded":"2023-09-30T00:17:51.962963+0000","last_fullsized":"2023-09-30T00:17:51.962963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:28.604134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"18'4","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960480+0000","last_change":"2023-09-30T00:17:51.590387+0000","last_active":"2023-09-30T00:17:51.960480+0000","last_peered":"2023-09-30T00:17:51.960480+0000","last_clean":"2023-09-30T00:17:51.960480+0000","last_became_active":"2023-09-30T00:17:51.585479+0000","last_became_peered":"2023-09-30T00:17:51.585479+0000","last_unstale":"2023-09-30T00:17:51.960480+0000","last_undegraded":"2023-09-30T00:17:51.960480+0000","last_fullsized":"2023-09-30T00:17:51.960480+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.959772+0000","last_change":"2023-09-30T00:17:51.949418+0000","last_active":"2023-09-30T00:17:51.959772+0000","last_peered":"2023-09-30T00:17:51.959772+0000","last_clean":"2023-09-30T00:17:51.959772+0000","last_became_active":"2023-09-30T00:17:51.949101+0000","last_became_peered":"2023-09-30T00:17:51.949101+0000","last_unstale":"2023-09-30T00:17:51.959772+0000","last_undegraded":"2023-09-30T00:17:51.959772+0000","last_fullsized":"2023-09-30T00:17:51.959772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975274+0000","last_change":"2023-09-30T00:17:51.964155+0000","last_active":"2023-09-30T00:17:51.975274+0000","last_peered":"2023-09-30T00:17:51.975274+0000","last_clean":"2023-09-30T00:17:51.975274+0000","last_became_active":"2023-09-30T00:17:51.958334+0000","last_became_peered":"2023-09-30T00:17:51.958334+0000","last_unstale":"2023-09-30T00:17:51.975274+0000","last_undegraded":"2023-09-30T00:17:51.975274+0000","last_fullsized":"2023-09-30T00:17:51.975274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.959645+0000","last_change":"2023-09-30T00:17:51.959066+0000","last_active":"2023-09-30T00:17:51.959645+0000","last_peered":"2023-09-30T00:17:51.959645+0000","last_clean":"2023-09-30T00:17:51.959645+0000","last_became_active":"2023-09-30T00:17:51.958741+0000","last_became_peered":"2023-09-30T00:17:51.958741+0000","last_unstale":"2023-09-30T00:17:51.959645+0000","last_undegraded":"2023-09-30T00:17:51.959645+0000","last_fullsized":"2023-09-30T00:17:51.959645+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:54:35.602823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976104+0000","last_change":"2023-09-30T00:17:51.970791+0000","last_active":"2023-09-30T00:17:51.976104+0000","last_peered":"2023-09-30T00:17:51.976104+0000","last_clean":"2023-09-30T00:17:51.976104+0000","last_became_active":"2023-09-30T00:17:51.964142+0000","last_became_peered":"2023-09-30T00:17:51.964142+0000","last_unstale":"2023-09-30T00:17:51.976104+0000","last_undegraded":"2023-09-30T00:17:51.976104+0000","last_fullsized":"2023-09-30T00:17:51.976104+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.967656+0000","last_change":"2023-09-30T00:17:51.967167+0000","last_active":"2023-09-30T00:17:51.967656+0000","last_peered":"2023-09-30T00:17:51.967656+0000","last_clean":"2023-09-30T00:17:51.967656+0000","last_became_active":"2023-09-30T00:17:51.966937+0000","last_became_peered":"2023-09-30T00:17:51.966937+0000","last_unstale":"2023-09-30T00:17:51.967656+0000","last_undegraded":"2023-09-30T00:17:51.967656+0000","last_fullsized":"2023-09-30T00:17:51.967656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:32:11.138421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960344+0000","last_change":"2023-09-30T00:17:51.959728+0000","last_active":"2023-09-30T00:17:51.960344+0000","last_peered":"2023-09-30T00:17:51.960344+0000","last_clean":"2023-09-30T00:17:51.960344+0000","last_became_active":"2023-09-30T00:17:51.959512+0000","last_became_peered":"2023-09-30T00:17:51.959512+0000","last_unstale":"2023-09-30T00:17:51.960344+0000","last_undegraded":"2023-09-30T00:17:51.960344+0000","last_fullsized":"2023-09-30T00:17:51.960344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:05:15.182292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672968,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9232,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362608,"statfs":{"total":96636764160,"available":96627310592,"internally_reserved":0,"allocated":794624,"data_stored":651535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]},{"osd":2,"up_from":10,"seq":42949672968,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8072,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363768,"statfs":{"total":96636764160,"available":96628498432,"internally_reserved":0,"allocated":196608,"data_stored":60391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9084,"kb_used_data":756,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94362756,"statfs":{"total":96636764160,"available":96627462144,"internally_reserved":0,"allocated":774144,"data_stored":650359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":1,"up_from":10,"seq":42949672968,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8064,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363776,"statfs":{"total":96636764160,"available":96628506624,"internally_reserved":0,"allocated":188416,"data_stored":60083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:18:05.668 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:18:05.816 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.816+0000 7f01c1fed700 1 Processor -- start 2023-09-30T00:18:05.817 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.816+0000 7f01c1fed700 1 -- start start 2023-09-30T00:18:05.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.816+0000 7f01c1fed700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01bc068590 0x7f01bc0a2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.816+0000 7f01c1fed700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc0a92d0 0x7f01bc0a3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.816+0000 7f01c1fed700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f01bc0a1140 0x7f01bc13da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.816+0000 7f01c1fed700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01bc0a26d0 con 0x7f01bc068590 2023-09-30T00:18:05.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.816+0000 7f01c1fed700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01bc068960 con 0x7f01bc0a1140 2023-09-30T00:18:05.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.816+0000 7f01c1fed700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc0a2ba0 con 0x7f01bc0a92d0 2023-09-30T00:18:05.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01c0feb700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01bc068590 0x7f01bc0a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01bbfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc0a92d0 0x7f01bc0a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01bbfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc0a92d0 0x7f01bc0a3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:37326/0 (socket says 172.21.15.94:37326) 2023-09-30T00:18:05.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01c0feb700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01bc068590 0x7f01bc0a2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36258/0 (socket says 172.21.15.94:36258) 2023-09-30T00:18:05.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01bbfff700 1 -- 172.21.15.94:0/1237420749 learned_addr learned my addr 172.21.15.94:0/1237420749 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:05.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01c17ec700 1 --2- 172.21.15.94:0/1237420749 >> v2:172.21.15.121:3300/0 conn(0x7f01bc0a1140 0x7f01bc13da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01bbfff700 1 -- 172.21.15.94:0/1237420749 >> v2:172.21.15.121:3300/0 conn(0x7f01bc0a1140 msgr2=0x7f01bc13da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:05.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01bbfff700 1 --2- 172.21.15.94:0/1237420749 >> v2:172.21.15.121:3300/0 conn(0x7f01bc0a1140 0x7f01bc13da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01bbfff700 1 -- 172.21.15.94:0/1237420749 >> v2:172.21.15.94:3300/0 conn(0x7f01bc068590 msgr2=0x7f01bc0a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:05.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01bbfff700 1 --2- 172.21.15.94:0/1237420749 >> v2:172.21.15.94:3300/0 conn(0x7f01bc068590 0x7f01bc0a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01bbfff700 1 -- 172.21.15.94:0/1237420749 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc0a28e0 con 0x7f01bc0a92d0 2023-09-30T00:18:05.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01c0feb700 1 --2- 172.21.15.94:0/1237420749 >> v2:172.21.15.94:3300/0 conn(0x7f01bc068590 0x7f01bc0a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:05.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01c17ec700 1 --2- 172.21.15.94:0/1237420749 >> v2:172.21.15.121:3300/0 conn(0x7f01bc0a1140 0x7f01bc13da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:05.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01bbfff700 1 --2- 172.21.15.94:0/1237420749 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc0a92d0 0x7f01bc0a3040 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f01bc06a690 tx=0x7f01b0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9edd0e3c0a31817d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01bb7fe700 1 -- 172.21.15.94:0/1237420749 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0009e70 con 0x7f01bc0a92d0 2023-09-30T00:18:05.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01bb7fe700 1 -- 172.21.15.94:0/1237420749 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b000b860 con 0x7f01bc0a92d0 2023-09-30T00:18:05.824 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.817+0000 7f01bb7fe700 1 -- 172.21.15.94:0/1237420749 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b000bc00 con 0x7f01bc0a92d0 2023-09-30T00:18:05.824 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.818+0000 7f01c1fed700 1 -- 172.21.15.94:0/1237420749 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc0a92d0 msgr2=0x7f01bc0a3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.824 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.818+0000 7f01c1fed700 1 --2- 172.21.15.94:0/1237420749 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc0a92d0 0x7f01bc0a3040 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f01bc06a690 tx=0x7f01b0009250 comp rx=0 tx=0).stop 2023-09-30T00:18:05.824 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.818+0000 7f01c1fed700 1 -- 172.21.15.94:0/1237420749 shutdown_connections 2023-09-30T00:18:05.824 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.818+0000 7f01c1fed700 1 --2- 172.21.15.94:0/1237420749 >> v2:172.21.15.121:3300/0 conn(0x7f01bc0a1140 0x7f01bc13da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.824 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.818+0000 7f01c1fed700 1 --2- 172.21.15.94:0/1237420749 >> v2:172.21.15.94:3300/0 conn(0x7f01bc068590 0x7f01bc0a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.824 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.818+0000 7f01c1fed700 1 --2- 172.21.15.94:0/1237420749 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc0a92d0 0x7f01bc0a3040 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.824 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.818+0000 7f01c1fed700 1 -- 172.21.15.94:0/1237420749 >> 172.21.15.94:0/1237420749 conn(0x7f01bc0a96f0 msgr2=0x7f01bc067c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:05.824 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.818+0000 7f01c1fed700 1 -- 172.21.15.94:0/1237420749 shutdown_connections 2023-09-30T00:18:05.824 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.818+0000 7f01c1fed700 1 -- 172.21.15.94:0/1237420749 wait complete. 2023-09-30T00:18:05.824 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.819+0000 7f01c1fed700 1 Processor -- start 2023-09-30T00:18:05.825 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.819+0000 7f01c1fed700 1 -- start start 2023-09-30T00:18:05.825 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.819+0000 7f01c1fed700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc068590 0x7f01bc0abe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.825 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.819+0000 7f01c1fed700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01bc0a1140 0x7f01bc0a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.825 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.819+0000 7f01c1fed700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01bc0ac360 0x7f01bc0a67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.825 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.819+0000 7f01c1fed700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc150360 con 0x7f01bc0a1140 2023-09-30T00:18:05.825 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.819+0000 7f01c1fed700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc0a5b90 con 0x7f01bc0ac360 2023-09-30T00:18:05.825 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.819+0000 7f01c1fed700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc0ab590 con 0x7f01bc068590 2023-09-30T00:18:05.825 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01c0feb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc068590 0x7f01bc0abe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.825 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01c0feb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc068590 0x7f01bc0abe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:37334/0 (socket says 172.21.15.94:37334) 2023-09-30T00:18:05.825 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01bbfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01bc0a1140 0x7f01bc0a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.825 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01c0feb700 1 -- 172.21.15.94:0/1013930917 learned_addr learned my addr 172.21.15.94:0/1013930917 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:05.825 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01c17ec700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01bc0ac360 0x7f01bc0a67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.826 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01c0feb700 1 -- 172.21.15.94:0/1013930917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01bc0ac360 msgr2=0x7f01bc0a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.826 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01c0feb700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01bc0ac360 0x7f01bc0a67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.826 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01c0feb700 1 -- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01bc0a1140 msgr2=0x7f01bc0a6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:05.826 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01c0feb700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01bc0a1140 0x7f01bc0a6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:05.826 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01c0feb700 1 -- 172.21.15.94:0/1013930917 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b4009a20 con 0x7f01bc068590 2023-09-30T00:18:05.826 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01c0feb700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc068590 0x7f01bc0abe20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f01b400c220 tx=0x7f01b400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.826 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01b8ff9700 1 -- 172.21.15.94:0/1013930917 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b400cdb0 con 0x7f01bc068590 2023-09-30T00:18:05.826 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01b8ff9700 1 -- 172.21.15.94:0/1013930917 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b400f040 con 0x7f01bc068590 2023-09-30T00:18:05.826 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01b8ff9700 1 -- 172.21.15.94:0/1013930917 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4014690 con 0x7f01bc068590 2023-09-30T00:18:05.826 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01c1fed700 1 -- 172.21.15.94:0/1013930917 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b00094f0 con 0x7f01bc068590 2023-09-30T00:18:05.826 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.820+0000 7f01c1fed700 1 -- 172.21.15.94:0/1013930917 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01bc144a80 con 0x7f01bc068590 2023-09-30T00:18:05.827 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.821+0000 7f01c1fed700 1 -- 172.21.15.94:0/1013930917 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01bc069340 con 0x7f01bc068590 2023-09-30T00:18:05.827 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.822+0000 7f01b8ff9700 1 -- 172.21.15.94:0/1013930917 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f01b4014a00 con 0x7f01bc068590 2023-09-30T00:18:05.828 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.822+0000 7f01b8ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:05.828 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.822+0000 7f01b8ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:05.828 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.822+0000 7f01b8ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:05.828 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.822+0000 7f01b8ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:05.828 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.822+0000 7f01b8ff9700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f01a8098780 0x7f01a809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:05.828 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.823+0000 7f01bbfff700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f01a8098780 0x7f01a809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:05.828 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.823+0000 7f01b8ff9700 1 -- 172.21.15.94:0/1013930917 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f01b40c1430 con 0x7f01bc068590 2023-09-30T00:18:05.828 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.823+0000 7f01bbfff700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f01a8098780 0x7f01a809ac30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f01bc06a690 tx=0x7f01b0017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:05.829 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:05.827+0000 7f01b8ff9700 1 -- 172.21.15.94:0/1013930917 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f01b408ea40 con 0x7f01bc068590 2023-09-30T00:18:06.004 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.004+0000 7f01c1fed700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2023-09-30T00:18:06.004 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.004+0000 7f01c1fed700 1 -- 172.21.15.94:0/1013930917 --> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01bc087180 con 0x7f01a8098780 2023-09-30T00:18:06.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.011+0000 7f01b8ff9700 1 -- 172.21.15.94:0/1013930917 <== mgr.4106 v2:172.21.15.94:6808/84242 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+269342 (secure 0 0 0) 0x7f01bc087180 con 0x7f01a8098780 2023-09-30T00:18:06.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.011+0000 7f01b8ff9700 20 mgrc handle_command_reply tid 0 r 0 2023-09-30T00:18:06.014 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:06.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 10 mgrc shutdown 2023-09-30T00:18:06.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 1 -- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f01a8098780 msgr2=0x7f01a809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:06.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f01a8098780 0x7f01a809ac30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f01bc06a690 tx=0x7f01b0017000 comp rx=0 tx=0).stop 2023-09-30T00:18:06.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 1 -- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc068590 msgr2=0x7f01bc0abe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:06.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc068590 0x7f01bc0abe20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f01b400c220 tx=0x7f01b400b040 comp rx=0 tx=0).stop 2023-09-30T00:18:06.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 1 -- 172.21.15.94:0/1013930917 shutdown_connections 2023-09-30T00:18:06.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f01a8098780 0x7f01a809ac30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01bc068590 0x7f01bc0abe20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01bc0ac360 0x7f01bc0a67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 1 --2- 172.21.15.94:0/1013930917 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01bc0a1140 0x7f01bc0a6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 1 -- 172.21.15.94:0/1013930917 >> 172.21.15.94:0/1013930917 conn(0x7f01bc0a96f0 msgr2=0x7f01bc13bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:06.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 1 -- 172.21.15.94:0/1013930917 shutdown_connections 2023-09-30T00:18:06.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.014+0000 7f01c1fed700 1 -- 172.21.15.94:0/1013930917 wait complete. 2023-09-30T00:18:06.018 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2023-09-30T00:18:06.026 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-09-30T00:18:05.853431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":34452,"kb_used_data":1908,"kb_used_omap":0,"kb_used_meta":32512,"kb_avail":377452908,"statfs":{"total":386547056640,"available":386511777792,"internally_reserved":0,"allocated":1953792,"data_stored":1422368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33292288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002610"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962397+0000","last_change":"2023-09-30T00:17:51.960193+0000","last_active":"2023-09-30T00:17:51.962397+0000","last_peered":"2023-09-30T00:17:51.962397+0000","last_clean":"2023-09-30T00:17:51.962397+0000","last_became_active":"2023-09-30T00:17:51.959888+0000","last_became_peered":"2023-09-30T00:17:51.959888+0000","last_unstale":"2023-09-30T00:17:51.962397+0000","last_undegraded":"2023-09-30T00:17:51.962397+0000","last_fullsized":"2023-09-30T00:17:51.962397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:40:42.458928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972435+0000","last_change":"2023-09-30T00:17:51.968577+0000","last_active":"2023-09-30T00:17:51.972435+0000","last_peered":"2023-09-30T00:17:51.972435+0000","last_clean":"2023-09-30T00:17:51.972435+0000","last_became_active":"2023-09-30T00:17:51.968275+0000","last_became_peered":"2023-09-30T00:17:51.968275+0000","last_unstale":"2023-09-30T00:17:51.972435+0000","last_undegraded":"2023-09-30T00:17:51.972435+0000","last_fullsized":"2023-09-30T00:17:51.972435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:19:24.974156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977703+0000","last_change":"2023-09-30T00:17:51.974359+0000","last_active":"2023-09-30T00:17:51.977703+0000","last_peered":"2023-09-30T00:17:51.977703+0000","last_clean":"2023-09-30T00:17:51.977703+0000","last_became_active":"2023-09-30T00:17:51.973789+0000","last_became_peered":"2023-09-30T00:17:51.973789+0000","last_unstale":"2023-09-30T00:17:51.977703+0000","last_undegraded":"2023-09-30T00:17:51.977703+0000","last_fullsized":"2023-09-30T00:17:51.977703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:41:27.523811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971859+0000","last_change":"2023-09-30T00:17:51.958127+0000","last_active":"2023-09-30T00:17:51.971859+0000","last_peered":"2023-09-30T00:17:51.971859+0000","last_clean":"2023-09-30T00:17:51.971859+0000","last_became_active":"2023-09-30T00:17:51.957850+0000","last_became_peered":"2023-09-30T00:17:51.957850+0000","last_unstale":"2023-09-30T00:17:51.971859+0000","last_undegraded":"2023-09-30T00:17:51.971859+0000","last_fullsized":"2023-09-30T00:17:51.971859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:48:36.950659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973474+0000","last_change":"2023-09-30T00:17:51.966792+0000","last_active":"2023-09-30T00:17:51.973474+0000","last_peered":"2023-09-30T00:17:51.973474+0000","last_clean":"2023-09-30T00:17:51.973474+0000","last_became_active":"2023-09-30T00:17:51.965887+0000","last_became_peered":"2023-09-30T00:17:51.965887+0000","last_unstale":"2023-09-30T00:17:51.973474+0000","last_undegraded":"2023-09-30T00:17:51.973474+0000","last_fullsized":"2023-09-30T00:17:51.973474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973636+0000","last_change":"2023-09-30T00:17:51.967175+0000","last_active":"2023-09-30T00:17:51.973636+0000","last_peered":"2023-09-30T00:17:51.973636+0000","last_clean":"2023-09-30T00:17:51.973636+0000","last_became_active":"2023-09-30T00:17:51.966408+0000","last_became_peered":"2023-09-30T00:17:51.966408+0000","last_unstale":"2023-09-30T00:17:51.973636+0000","last_undegraded":"2023-09-30T00:17:51.973636+0000","last_fullsized":"2023-09-30T00:17:51.973636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:47:56.029102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972783+0000","last_change":"2023-09-30T00:17:51.964264+0000","last_active":"2023-09-30T00:17:51.972783+0000","last_peered":"2023-09-30T00:17:51.972783+0000","last_clean":"2023-09-30T00:17:51.972783+0000","last_became_active":"2023-09-30T00:17:51.962740+0000","last_became_peered":"2023-09-30T00:17:51.962740+0000","last_unstale":"2023-09-30T00:17:51.972783+0000","last_undegraded":"2023-09-30T00:17:51.972783+0000","last_fullsized":"2023-09-30T00:17:51.972783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:52:26.654463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975296+0000","last_change":"2023-09-30T00:17:51.967599+0000","last_active":"2023-09-30T00:17:51.975296+0000","last_peered":"2023-09-30T00:17:51.975296+0000","last_clean":"2023-09-30T00:17:51.975296+0000","last_became_active":"2023-09-30T00:17:51.961762+0000","last_became_peered":"2023-09-30T00:17:51.961762+0000","last_unstale":"2023-09-30T00:17:51.975296+0000","last_undegraded":"2023-09-30T00:17:51.975296+0000","last_fullsized":"2023-09-30T00:17:51.975296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:35:31.620523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971780+0000","last_change":"2023-09-30T00:17:51.971131+0000","last_active":"2023-09-30T00:17:51.971780+0000","last_peered":"2023-09-30T00:17:51.971780+0000","last_clean":"2023-09-30T00:17:51.971780+0000","last_became_active":"2023-09-30T00:17:51.970865+0000","last_became_peered":"2023-09-30T00:17:51.970865+0000","last_unstale":"2023-09-30T00:17:51.971780+0000","last_undegraded":"2023-09-30T00:17:51.971780+0000","last_fullsized":"2023-09-30T00:17:51.971780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:53:33.380570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972400+0000","last_change":"2023-09-30T00:17:51.963624+0000","last_active":"2023-09-30T00:17:51.972400+0000","last_peered":"2023-09-30T00:17:51.972400+0000","last_clean":"2023-09-30T00:17:51.972400+0000","last_became_active":"2023-09-30T00:17:51.963303+0000","last_became_peered":"2023-09-30T00:17:51.963303+0000","last_unstale":"2023-09-30T00:17:51.972400+0000","last_undegraded":"2023-09-30T00:17:51.972400+0000","last_fullsized":"2023-09-30T00:17:51.972400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971954+0000","last_change":"2023-09-30T00:17:51.961620+0000","last_active":"2023-09-30T00:17:51.971954+0000","last_peered":"2023-09-30T00:17:51.971954+0000","last_clean":"2023-09-30T00:17:51.971954+0000","last_became_active":"2023-09-30T00:17:51.958237+0000","last_became_peered":"2023-09-30T00:17:51.958237+0000","last_unstale":"2023-09-30T00:17:51.971954+0000","last_undegraded":"2023-09-30T00:17:51.971954+0000","last_fullsized":"2023-09-30T00:17:51.971954+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977575+0000","last_change":"2023-09-30T00:17:51.974397+0000","last_active":"2023-09-30T00:17:51.977575+0000","last_peered":"2023-09-30T00:17:51.977575+0000","last_clean":"2023-09-30T00:17:51.977575+0000","last_became_active":"2023-09-30T00:17:51.974025+0000","last_became_peered":"2023-09-30T00:17:51.974025+0000","last_unstale":"2023-09-30T00:17:51.977575+0000","last_undegraded":"2023-09-30T00:17:51.977575+0000","last_fullsized":"2023-09-30T00:17:51.977575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:49:07.723751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.966627+0000","last_change":"2023-09-30T00:17:51.966187+0000","last_active":"2023-09-30T00:17:51.966627+0000","last_peered":"2023-09-30T00:17:51.966627+0000","last_clean":"2023-09-30T00:17:51.966627+0000","last_became_active":"2023-09-30T00:17:51.965900+0000","last_became_peered":"2023-09-30T00:17:51.965900+0000","last_unstale":"2023-09-30T00:17:51.966627+0000","last_undegraded":"2023-09-30T00:17:51.966627+0000","last_fullsized":"2023-09-30T00:17:51.966627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:30:07.208946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976147+0000","last_change":"2023-09-30T00:17:51.970985+0000","last_active":"2023-09-30T00:17:51.976147+0000","last_peered":"2023-09-30T00:17:51.976147+0000","last_clean":"2023-09-30T00:17:51.976147+0000","last_became_active":"2023-09-30T00:17:51.966979+0000","last_became_peered":"2023-09-30T00:17:51.966979+0000","last_unstale":"2023-09-30T00:17:51.976147+0000","last_undegraded":"2023-09-30T00:17:51.976147+0000","last_fullsized":"2023-09-30T00:17:51.976147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:09:28.833739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975911+0000","last_change":"2023-09-30T00:17:51.970398+0000","last_active":"2023-09-30T00:17:51.975911+0000","last_peered":"2023-09-30T00:17:51.975911+0000","last_clean":"2023-09-30T00:17:51.975911+0000","last_became_active":"2023-09-30T00:17:51.961675+0000","last_became_peered":"2023-09-30T00:17:51.961675+0000","last_unstale":"2023-09-30T00:17:51.975911+0000","last_undegraded":"2023-09-30T00:17:51.975911+0000","last_fullsized":"2023-09-30T00:17:51.975911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:33:21.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972718+0000","last_change":"2023-09-30T00:17:51.969081+0000","last_active":"2023-09-30T00:17:51.972718+0000","last_peered":"2023-09-30T00:17:51.972718+0000","last_clean":"2023-09-30T00:17:51.972718+0000","last_became_active":"2023-09-30T00:17:51.966447+0000","last_became_peered":"2023-09-30T00:17:51.966447+0000","last_unstale":"2023-09-30T00:17:51.972718+0000","last_undegraded":"2023-09-30T00:17:51.972718+0000","last_fullsized":"2023-09-30T00:17:51.972718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:19:12.188711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960350+0000","last_change":"2023-09-30T00:17:51.953980+0000","last_active":"2023-09-30T00:17:51.960350+0000","last_peered":"2023-09-30T00:17:51.960350+0000","last_clean":"2023-09-30T00:17:51.960350+0000","last_became_active":"2023-09-30T00:17:51.952191+0000","last_became_peered":"2023-09-30T00:17:51.952191+0000","last_unstale":"2023-09-30T00:17:51.960350+0000","last_undegraded":"2023-09-30T00:17:51.960350+0000","last_fullsized":"2023-09-30T00:17:51.960350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:52:46.774408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973554+0000","last_change":"2023-09-30T00:17:51.969589+0000","last_active":"2023-09-30T00:17:51.973554+0000","last_peered":"2023-09-30T00:17:51.973554+0000","last_clean":"2023-09-30T00:17:51.973554+0000","last_became_active":"2023-09-30T00:17:51.967300+0000","last_became_peered":"2023-09-30T00:17:51.967300+0000","last_unstale":"2023-09-30T00:17:51.973554+0000","last_undegraded":"2023-09-30T00:17:51.973554+0000","last_fullsized":"2023-09-30T00:17:51.973554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:22:57.623135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.966670+0000","last_change":"2023-09-30T00:17:51.966047+0000","last_active":"2023-09-30T00:17:51.966670+0000","last_peered":"2023-09-30T00:17:51.966670+0000","last_clean":"2023-09-30T00:17:51.966670+0000","last_became_active":"2023-09-30T00:17:51.965457+0000","last_became_peered":"2023-09-30T00:17:51.965457+0000","last_unstale":"2023-09-30T00:17:51.966670+0000","last_undegraded":"2023-09-30T00:17:51.966670+0000","last_fullsized":"2023-09-30T00:17:51.966670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:24:33.105806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972047+0000","last_change":"2023-09-30T00:17:51.961502+0000","last_active":"2023-09-30T00:17:51.972047+0000","last_peered":"2023-09-30T00:17:51.972047+0000","last_clean":"2023-09-30T00:17:51.972047+0000","last_became_active":"2023-09-30T00:17:51.961191+0000","last_became_peered":"2023-09-30T00:17:51.961191+0000","last_unstale":"2023-09-30T00:17:51.972047+0000","last_undegraded":"2023-09-30T00:17:51.972047+0000","last_fullsized":"2023-09-30T00:17:51.972047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:05:37.196886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976121+0000","last_change":"2023-09-30T00:17:51.970789+0000","last_active":"2023-09-30T00:17:51.976121+0000","last_peered":"2023-09-30T00:17:51.976121+0000","last_clean":"2023-09-30T00:17:51.976121+0000","last_became_active":"2023-09-30T00:17:51.967616+0000","last_became_peered":"2023-09-30T00:17:51.967616+0000","last_unstale":"2023-09-30T00:17:51.976121+0000","last_undegraded":"2023-09-30T00:17:51.976121+0000","last_fullsized":"2023-09-30T00:17:51.976121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976127+0000","last_change":"2023-09-30T00:17:51.970797+0000","last_active":"2023-09-30T00:17:51.976127+0000","last_peered":"2023-09-30T00:17:51.976127+0000","last_clean":"2023-09-30T00:17:51.976127+0000","last_became_active":"2023-09-30T00:17:51.963382+0000","last_became_peered":"2023-09-30T00:17:51.963382+0000","last_unstale":"2023-09-30T00:17:51.976127+0000","last_undegraded":"2023-09-30T00:17:51.976127+0000","last_fullsized":"2023-09-30T00:17:51.976127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:35:22.077360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972764+0000","last_change":"2023-09-30T00:17:51.963058+0000","last_active":"2023-09-30T00:17:51.972764+0000","last_peered":"2023-09-30T00:17:51.972764+0000","last_clean":"2023-09-30T00:17:51.972764+0000","last_became_active":"2023-09-30T00:17:51.959450+0000","last_became_peered":"2023-09-30T00:17:51.959450+0000","last_unstale":"2023-09-30T00:17:51.972764+0000","last_undegraded":"2023-09-30T00:17:51.972764+0000","last_fullsized":"2023-09-30T00:17:51.972764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:41:36.585190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975442+0000","last_change":"2023-09-30T00:17:51.967581+0000","last_active":"2023-09-30T00:17:51.975442+0000","last_peered":"2023-09-30T00:17:51.975442+0000","last_clean":"2023-09-30T00:17:51.975442+0000","last_became_active":"2023-09-30T00:17:51.961348+0000","last_became_peered":"2023-09-30T00:17:51.961348+0000","last_unstale":"2023-09-30T00:17:51.975442+0000","last_undegraded":"2023-09-30T00:17:51.975442+0000","last_fullsized":"2023-09-30T00:17:51.975442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:11:09.140469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975871+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.975871+0000","last_peered":"2023-09-30T00:17:51.975871+0000","last_clean":"2023-09-30T00:17:51.975871+0000","last_became_active":"2023-09-30T00:17:51.963534+0000","last_became_peered":"2023-09-30T00:17:51.963534+0000","last_unstale":"2023-09-30T00:17:51.975871+0000","last_undegraded":"2023-09-30T00:17:51.975871+0000","last_fullsized":"2023-09-30T00:17:51.975871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:21:49.086269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.961574+0000","last_change":"2023-09-30T00:17:51.961210+0000","last_active":"2023-09-30T00:17:51.961574+0000","last_peered":"2023-09-30T00:17:51.961574+0000","last_clean":"2023-09-30T00:17:51.961574+0000","last_became_active":"2023-09-30T00:17:51.961010+0000","last_became_peered":"2023-09-30T00:17:51.961010+0000","last_unstale":"2023-09-30T00:17:51.961574+0000","last_undegraded":"2023-09-30T00:17:51.961574+0000","last_fullsized":"2023-09-30T00:17:51.961574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:31:50.854220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972810+0000","last_change":"2023-09-30T00:17:51.968803+0000","last_active":"2023-09-30T00:17:51.972810+0000","last_peered":"2023-09-30T00:17:51.972810+0000","last_clean":"2023-09-30T00:17:51.972810+0000","last_became_active":"2023-09-30T00:17:51.965833+0000","last_became_peered":"2023-09-30T00:17:51.965833+0000","last_unstale":"2023-09-30T00:17:51.972810+0000","last_undegraded":"2023-09-30T00:17:51.972810+0000","last_fullsized":"2023-09-30T00:17:51.972810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:30.780158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960606+0000","last_change":"2023-09-30T00:17:51.592497+0000","last_active":"2023-09-30T00:17:51.960606+0000","last_peered":"2023-09-30T00:17:51.960606+0000","last_clean":"2023-09-30T00:17:51.960606+0000","last_became_active":"2023-09-30T00:17:51.585128+0000","last_became_peered":"2023-09-30T00:17:51.585128+0000","last_unstale":"2023-09-30T00:17:51.960606+0000","last_undegraded":"2023-09-30T00:17:51.960606+0000","last_fullsized":"2023-09-30T00:17:51.960606+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:39:46.058014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977744+0000","last_change":"2023-09-30T00:17:51.974520+0000","last_active":"2023-09-30T00:17:51.977744+0000","last_peered":"2023-09-30T00:17:51.977744+0000","last_clean":"2023-09-30T00:17:51.977744+0000","last_became_active":"2023-09-30T00:17:51.974109+0000","last_became_peered":"2023-09-30T00:17:51.974109+0000","last_unstale":"2023-09-30T00:17:51.977744+0000","last_undegraded":"2023-09-30T00:17:51.977744+0000","last_fullsized":"2023-09-30T00:17:51.977744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:25:05.134336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941011+0000","last_change":"2023-09-30T00:17:51.587986+0000","last_active":"2023-09-30T00:17:51.941011+0000","last_peered":"2023-09-30T00:17:51.941011+0000","last_clean":"2023-09-30T00:17:51.941011+0000","last_became_active":"2023-09-30T00:17:51.587724+0000","last_became_peered":"2023-09-30T00:17:51.587724+0000","last_unstale":"2023-09-30T00:17:51.941011+0000","last_undegraded":"2023-09-30T00:17:51.941011+0000","last_fullsized":"2023-09-30T00:17:51.941011+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971773+0000","last_change":"2023-09-30T00:17:51.971105+0000","last_active":"2023-09-30T00:17:51.971773+0000","last_peered":"2023-09-30T00:17:51.971773+0000","last_clean":"2023-09-30T00:17:51.971773+0000","last_became_active":"2023-09-30T00:17:51.970846+0000","last_became_peered":"2023-09-30T00:17:51.970846+0000","last_unstale":"2023-09-30T00:17:51.971773+0000","last_undegraded":"2023-09-30T00:17:51.971773+0000","last_fullsized":"2023-09-30T00:17:51.971773+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:13:51.108522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952700+0000","last_change":"2023-09-30T00:17:51.596511+0000","last_active":"2023-09-30T00:17:51.952700+0000","last_peered":"2023-09-30T00:17:51.952700+0000","last_clean":"2023-09-30T00:17:51.952700+0000","last_became_active":"2023-09-30T00:17:51.596034+0000","last_became_peered":"2023-09-30T00:17:51.596034+0000","last_unstale":"2023-09-30T00:17:51.952700+0000","last_undegraded":"2023-09-30T00:17:51.952700+0000","last_fullsized":"2023-09-30T00:17:51.952700+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:44:29.988610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969658+0000","last_change":"2023-09-30T00:17:51.964766+0000","last_active":"2023-09-30T00:17:51.969658+0000","last_peered":"2023-09-30T00:17:51.969658+0000","last_clean":"2023-09-30T00:17:51.969658+0000","last_became_active":"2023-09-30T00:17:51.964496+0000","last_became_peered":"2023-09-30T00:17:51.964496+0000","last_unstale":"2023-09-30T00:17:51.969658+0000","last_undegraded":"2023-09-30T00:17:51.969658+0000","last_fullsized":"2023-09-30T00:17:51.969658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:51:41.317406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.942081+0000","last_change":"2023-09-30T00:17:51.577022+0000","last_active":"2023-09-30T00:17:51.942081+0000","last_peered":"2023-09-30T00:17:51.942081+0000","last_clean":"2023-09-30T00:17:51.942081+0000","last_became_active":"2023-09-30T00:17:51.576726+0000","last_became_peered":"2023-09-30T00:17:51.576726+0000","last_unstale":"2023-09-30T00:17:51.942081+0000","last_undegraded":"2023-09-30T00:17:51.942081+0000","last_fullsized":"2023-09-30T00:17:51.942081+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972739+0000","last_change":"2023-09-30T00:17:51.969564+0000","last_active":"2023-09-30T00:17:51.972739+0000","last_peered":"2023-09-30T00:17:51.972739+0000","last_clean":"2023-09-30T00:17:51.972739+0000","last_became_active":"2023-09-30T00:17:51.969240+0000","last_became_peered":"2023-09-30T00:17:51.969240+0000","last_unstale":"2023-09-30T00:17:51.972739+0000","last_undegraded":"2023-09-30T00:17:51.972739+0000","last_fullsized":"2023-09-30T00:17:51.972739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:42.701890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.949738+0000","last_change":"2023-09-30T00:17:51.588716+0000","last_active":"2023-09-30T00:17:51.949738+0000","last_peered":"2023-09-30T00:17:51.949738+0000","last_clean":"2023-09-30T00:17:51.949738+0000","last_became_active":"2023-09-30T00:17:51.588454+0000","last_became_peered":"2023-09-30T00:17:51.588454+0000","last_unstale":"2023-09-30T00:17:51.949738+0000","last_undegraded":"2023-09-30T00:17:51.949738+0000","last_fullsized":"2023-09-30T00:17:51.949738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.958204+0000","last_change":"2023-09-30T00:17:51.957598+0000","last_active":"2023-09-30T00:17:51.958204+0000","last_peered":"2023-09-30T00:17:51.958204+0000","last_clean":"2023-09-30T00:17:51.958204+0000","last_became_active":"2023-09-30T00:17:51.957336+0000","last_became_peered":"2023-09-30T00:17:51.957336+0000","last_unstale":"2023-09-30T00:17:51.958204+0000","last_undegraded":"2023-09-30T00:17:51.958204+0000","last_fullsized":"2023-09-30T00:17:51.958204+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:02:32.124291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"18'3","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973987+0000","last_change":"2023-09-30T00:17:51.589988+0000","last_active":"2023-09-30T00:17:51.973987+0000","last_peered":"2023-09-30T00:17:51.973987+0000","last_clean":"2023-09-30T00:17:51.973987+0000","last_became_active":"2023-09-30T00:17:51.589729+0000","last_became_peered":"2023-09-30T00:17:51.589729+0000","last_unstale":"2023-09-30T00:17:51.973987+0000","last_undegraded":"2023-09-30T00:17:51.973987+0000","last_fullsized":"2023-09-30T00:17:51.973987+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975239+0000","last_change":"2023-09-30T00:17:51.967245+0000","last_active":"2023-09-30T00:17:51.975239+0000","last_peered":"2023-09-30T00:17:51.975239+0000","last_clean":"2023-09-30T00:17:51.975239+0000","last_became_active":"2023-09-30T00:17:51.963755+0000","last_became_peered":"2023-09-30T00:17:51.963755+0000","last_unstale":"2023-09-30T00:17:51.975239+0000","last_undegraded":"2023-09-30T00:17:51.975239+0000","last_fullsized":"2023-09-30T00:17:51.975239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:57:31.735059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"18'6","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.983073+0000","last_change":"2023-09-30T00:17:51.592307+0000","last_active":"2023-09-30T00:17:51.983073+0000","last_peered":"2023-09-30T00:17:51.983073+0000","last_clean":"2023-09-30T00:17:51.983073+0000","last_became_active":"2023-09-30T00:17:51.591753+0000","last_became_peered":"2023-09-30T00:17:51.591753+0000","last_unstale":"2023-09-30T00:17:51.983073+0000","last_undegraded":"2023-09-30T00:17:51.983073+0000","last_fullsized":"2023-09-30T00:17:51.983073+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:14:46.193926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968817+0000","last_change":"2023-09-30T00:17:51.967859+0000","last_active":"2023-09-30T00:17:51.968817+0000","last_peered":"2023-09-30T00:17:51.968817+0000","last_clean":"2023-09-30T00:17:51.968817+0000","last_became_active":"2023-09-30T00:17:51.967118+0000","last_became_peered":"2023-09-30T00:17:51.967118+0000","last_unstale":"2023-09-30T00:17:51.968817+0000","last_undegraded":"2023-09-30T00:17:51.968817+0000","last_fullsized":"2023-09-30T00:17:51.968817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:50.694814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.969722+0000","last_change":"2023-09-30T00:17:51.591516+0000","last_active":"2023-09-30T00:17:51.969722+0000","last_peered":"2023-09-30T00:17:51.969722+0000","last_clean":"2023-09-30T00:17:51.969722+0000","last_became_active":"2023-09-30T00:17:51.591206+0000","last_became_peered":"2023-09-30T00:17:51.591206+0000","last_unstale":"2023-09-30T00:17:51.969722+0000","last_undegraded":"2023-09-30T00:17:51.969722+0000","last_fullsized":"2023-09-30T00:17:51.969722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:22.807978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977754+0000","last_change":"2023-09-30T00:17:51.974628+0000","last_active":"2023-09-30T00:17:51.977754+0000","last_peered":"2023-09-30T00:17:51.977754+0000","last_clean":"2023-09-30T00:17:51.977754+0000","last_became_active":"2023-09-30T00:17:51.974276+0000","last_became_peered":"2023-09-30T00:17:51.974276+0000","last_unstale":"2023-09-30T00:17:51.977754+0000","last_undegraded":"2023-09-30T00:17:51.977754+0000","last_fullsized":"2023-09-30T00:17:51.977754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:08:16.715786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.946238+0000","last_change":"2023-09-30T00:17:51.589138+0000","last_active":"2023-09-30T00:17:51.946238+0000","last_peered":"2023-09-30T00:17:51.946238+0000","last_clean":"2023-09-30T00:17:51.946238+0000","last_became_active":"2023-09-30T00:17:51.588883+0000","last_became_peered":"2023-09-30T00:17:51.588883+0000","last_unstale":"2023-09-30T00:17:51.946238+0000","last_undegraded":"2023-09-30T00:17:51.946238+0000","last_fullsized":"2023-09-30T00:17:51.946238+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977625+0000","last_change":"2023-09-30T00:17:51.972690+0000","last_active":"2023-09-30T00:17:51.977625+0000","last_peered":"2023-09-30T00:17:51.977625+0000","last_clean":"2023-09-30T00:17:51.977625+0000","last_became_active":"2023-09-30T00:17:51.969872+0000","last_became_peered":"2023-09-30T00:17:51.969872+0000","last_unstale":"2023-09-30T00:17:51.977625+0000","last_undegraded":"2023-09-30T00:17:51.977625+0000","last_fullsized":"2023-09-30T00:17:51.977625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952325+0000","last_change":"2023-09-30T00:17:51.597410+0000","last_active":"2023-09-30T00:17:51.952325+0000","last_peered":"2023-09-30T00:17:51.952325+0000","last_clean":"2023-09-30T00:17:51.952325+0000","last_became_active":"2023-09-30T00:17:51.597168+0000","last_became_peered":"2023-09-30T00:17:51.597168+0000","last_unstale":"2023-09-30T00:17:51.952325+0000","last_undegraded":"2023-09-30T00:17:51.952325+0000","last_fullsized":"2023-09-30T00:17:51.952325+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:44:45.339994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973574+0000","last_change":"2023-09-30T00:17:51.969752+0000","last_active":"2023-09-30T00:17:51.973574+0000","last_peered":"2023-09-30T00:17:51.973574+0000","last_clean":"2023-09-30T00:17:51.973574+0000","last_became_active":"2023-09-30T00:17:51.968993+0000","last_became_peered":"2023-09-30T00:17:51.968993+0000","last_unstale":"2023-09-30T00:17:51.973574+0000","last_undegraded":"2023-09-30T00:17:51.973574+0000","last_fullsized":"2023-09-30T00:17:51.973574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"18'8","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974927+0000","last_change":"2023-09-30T00:17:51.592339+0000","last_active":"2023-09-30T00:17:51.974927+0000","last_peered":"2023-09-30T00:17:51.974927+0000","last_clean":"2023-09-30T00:17:51.974927+0000","last_became_active":"2023-09-30T00:17:51.592134+0000","last_became_peered":"2023-09-30T00:17:51.592134+0000","last_unstale":"2023-09-30T00:17:51.974927+0000","last_undegraded":"2023-09-30T00:17:51.974927+0000","last_fullsized":"2023-09-30T00:17:51.974927+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:40:03.662954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.959394+0000","last_change":"2023-09-30T00:17:51.958723+0000","last_active":"2023-09-30T00:17:51.959394+0000","last_peered":"2023-09-30T00:17:51.959394+0000","last_clean":"2023-09-30T00:17:51.959394+0000","last_became_active":"2023-09-30T00:17:51.958465+0000","last_became_peered":"2023-09-30T00:17:51.958465+0000","last_unstale":"2023-09-30T00:17:51.959394+0000","last_undegraded":"2023-09-30T00:17:51.959394+0000","last_fullsized":"2023-09-30T00:17:51.959394+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:59:59.352777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968301+0000","last_change":"2023-09-30T00:17:51.577740+0000","last_active":"2023-09-30T00:17:51.968301+0000","last_peered":"2023-09-30T00:17:51.968301+0000","last_clean":"2023-09-30T00:17:51.968301+0000","last_became_active":"2023-09-30T00:17:51.577449+0000","last_became_peered":"2023-09-30T00:17:51.577449+0000","last_unstale":"2023-09-30T00:17:51.968301+0000","last_undegraded":"2023-09-30T00:17:51.968301+0000","last_fullsized":"2023-09-30T00:17:51.968301+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:49.948918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977756+0000","last_change":"2023-09-30T00:17:51.974425+0000","last_active":"2023-09-30T00:17:51.977756+0000","last_peered":"2023-09-30T00:17:51.977756+0000","last_clean":"2023-09-30T00:17:51.977756+0000","last_became_active":"2023-09-30T00:17:51.974143+0000","last_became_peered":"2023-09-30T00:17:51.974143+0000","last_unstale":"2023-09-30T00:17:51.977756+0000","last_undegraded":"2023-09-30T00:17:51.977756+0000","last_fullsized":"2023-09-30T00:17:51.977756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:04:54.968664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.972091+0000","last_change":"2023-09-30T00:17:51.578710+0000","last_active":"2023-09-30T00:17:51.972091+0000","last_peered":"2023-09-30T00:17:51.972091+0000","last_clean":"2023-09-30T00:17:51.972091+0000","last_became_active":"2023-09-30T00:17:51.578450+0000","last_became_peered":"2023-09-30T00:17:51.578450+0000","last_unstale":"2023-09-30T00:17:51.972091+0000","last_undegraded":"2023-09-30T00:17:51.972091+0000","last_fullsized":"2023-09-30T00:17:51.972091+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:05:53.462012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.958310+0000","last_change":"2023-09-30T00:17:51.951367+0000","last_active":"2023-09-30T00:17:51.958310+0000","last_peered":"2023-09-30T00:17:51.958310+0000","last_clean":"2023-09-30T00:17:51.958310+0000","last_became_active":"2023-09-30T00:17:51.951031+0000","last_became_peered":"2023-09-30T00:17:51.951031+0000","last_unstale":"2023-09-30T00:17:51.958310+0000","last_undegraded":"2023-09-30T00:17:51.958310+0000","last_fullsized":"2023-09-30T00:17:51.958310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976462+0000","last_change":"2023-09-30T00:17:51.594234+0000","last_active":"2023-09-30T00:17:51.976462+0000","last_peered":"2023-09-30T00:17:51.976462+0000","last_clean":"2023-09-30T00:17:51.976462+0000","last_became_active":"2023-09-30T00:17:51.594049+0000","last_became_peered":"2023-09-30T00:17:51.594049+0000","last_unstale":"2023-09-30T00:17:51.976462+0000","last_undegraded":"2023-09-30T00:17:51.976462+0000","last_fullsized":"2023-09-30T00:17:51.976462+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:53:48.206055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968844+0000","last_change":"2023-09-30T00:17:51.965298+0000","last_active":"2023-09-30T00:17:51.968844+0000","last_peered":"2023-09-30T00:17:51.968844+0000","last_clean":"2023-09-30T00:17:51.968844+0000","last_became_active":"2023-09-30T00:17:51.964965+0000","last_became_peered":"2023-09-30T00:17:51.964965+0000","last_unstale":"2023-09-30T00:17:51.968844+0000","last_undegraded":"2023-09-30T00:17:51.968844+0000","last_fullsized":"2023-09-30T00:17:51.968844+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:33:20.583162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.955628+0000","last_change":"2023-09-30T00:17:51.592485+0000","last_active":"2023-09-30T00:17:51.955628+0000","last_peered":"2023-09-30T00:17:51.955628+0000","last_clean":"2023-09-30T00:17:51.955628+0000","last_became_active":"2023-09-30T00:17:51.592054+0000","last_became_peered":"2023-09-30T00:17:51.592054+0000","last_unstale":"2023-09-30T00:17:51.955628+0000","last_undegraded":"2023-09-30T00:17:51.955628+0000","last_fullsized":"2023-09-30T00:17:51.955628+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:46:52.872573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.961561+0000","last_change":"2023-09-30T00:17:51.952057+0000","last_active":"2023-09-30T00:17:51.961561+0000","last_peered":"2023-09-30T00:17:51.961561+0000","last_clean":"2023-09-30T00:17:51.961561+0000","last_became_active":"2023-09-30T00:17:51.951537+0000","last_became_peered":"2023-09-30T00:17:51.951537+0000","last_unstale":"2023-09-30T00:17:51.961561+0000","last_undegraded":"2023-09-30T00:17:51.961561+0000","last_fullsized":"2023-09-30T00:17:51.961561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:20:25.199612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.966041+0000","last_change":"2023-09-30T00:17:51.578071+0000","last_active":"2023-09-30T00:17:51.966041+0000","last_peered":"2023-09-30T00:17:51.966041+0000","last_clean":"2023-09-30T00:17:51.966041+0000","last_became_active":"2023-09-30T00:17:51.577808+0000","last_became_peered":"2023-09-30T00:17:51.577808+0000","last_unstale":"2023-09-30T00:17:51.966041+0000","last_undegraded":"2023-09-30T00:17:51.966041+0000","last_fullsized":"2023-09-30T00:17:51.966041+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:20.626316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976183+0000","last_change":"2023-09-30T00:17:51.970822+0000","last_active":"2023-09-30T00:17:51.976183+0000","last_peered":"2023-09-30T00:17:51.976183+0000","last_clean":"2023-09-30T00:17:51.976183+0000","last_became_active":"2023-09-30T00:17:51.970520+0000","last_became_peered":"2023-09-30T00:17:51.970520+0000","last_unstale":"2023-09-30T00:17:51.976183+0000","last_undegraded":"2023-09-30T00:17:51.976183+0000","last_fullsized":"2023-09-30T00:17:51.976183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:07.347359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952250+0000","last_change":"2023-09-30T00:17:51.595723+0000","last_active":"2023-09-30T00:17:51.952250+0000","last_peered":"2023-09-30T00:17:51.952250+0000","last_clean":"2023-09-30T00:17:51.952250+0000","last_became_active":"2023-09-30T00:17:51.595473+0000","last_became_peered":"2023-09-30T00:17:51.595473+0000","last_unstale":"2023-09-30T00:17:51.952250+0000","last_undegraded":"2023-09-30T00:17:51.952250+0000","last_fullsized":"2023-09-30T00:17:51.952250+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:45:06.343890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980343+0000","last_change":"2023-09-30T00:17:51.977010+0000","last_active":"2023-09-30T00:17:51.980343+0000","last_peered":"2023-09-30T00:17:51.980343+0000","last_clean":"2023-09-30T00:17:51.980343+0000","last_became_active":"2023-09-30T00:17:51.976744+0000","last_became_peered":"2023-09-30T00:17:51.976744+0000","last_unstale":"2023-09-30T00:17:51.980343+0000","last_undegraded":"2023-09-30T00:17:51.980343+0000","last_fullsized":"2023-09-30T00:17:51.980343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:02:14.797729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952109+0000","last_change":"2023-09-30T00:17:51.595305+0000","last_active":"2023-09-30T00:17:51.952109+0000","last_peered":"2023-09-30T00:17:51.952109+0000","last_clean":"2023-09-30T00:17:51.952109+0000","last_became_active":"2023-09-30T00:17:51.595042+0000","last_became_peered":"2023-09-30T00:17:51.595042+0000","last_unstale":"2023-09-30T00:17:51.952109+0000","last_undegraded":"2023-09-30T00:17:51.952109+0000","last_fullsized":"2023-09-30T00:17:51.952109+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:54.681232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968754+0000","last_change":"2023-09-30T00:17:51.965816+0000","last_active":"2023-09-30T00:17:51.968754+0000","last_peered":"2023-09-30T00:17:51.968754+0000","last_clean":"2023-09-30T00:17:51.968754+0000","last_became_active":"2023-09-30T00:17:51.965549+0000","last_became_peered":"2023-09-30T00:17:51.965549+0000","last_unstale":"2023-09-30T00:17:51.968754+0000","last_undegraded":"2023-09-30T00:17:51.968754+0000","last_fullsized":"2023-09-30T00:17:51.968754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:36:30.526216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.953542+0000","last_change":"2023-09-30T00:17:51.596289+0000","last_active":"2023-09-30T00:17:51.953542+0000","last_peered":"2023-09-30T00:17:51.953542+0000","last_clean":"2023-09-30T00:17:51.953542+0000","last_became_active":"2023-09-30T00:17:51.595689+0000","last_became_peered":"2023-09-30T00:17:51.595689+0000","last_unstale":"2023-09-30T00:17:51.953542+0000","last_undegraded":"2023-09-30T00:17:51.953542+0000","last_fullsized":"2023-09-30T00:17:51.953542+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:46:55.016362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975889+0000","last_change":"2023-09-30T00:17:51.970420+0000","last_active":"2023-09-30T00:17:51.975889+0000","last_peered":"2023-09-30T00:17:51.975889+0000","last_clean":"2023-09-30T00:17:51.975889+0000","last_became_active":"2023-09-30T00:17:51.963999+0000","last_became_peered":"2023-09-30T00:17:51.963999+0000","last_unstale":"2023-09-30T00:17:51.975889+0000","last_undegraded":"2023-09-30T00:17:51.975889+0000","last_fullsized":"2023-09-30T00:17:51.975889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:55:34.567827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962989+0000","last_change":"2023-09-30T00:17:51.590485+0000","last_active":"2023-09-30T00:17:51.962989+0000","last_peered":"2023-09-30T00:17:51.962989+0000","last_clean":"2023-09-30T00:17:51.962989+0000","last_became_active":"2023-09-30T00:17:51.590210+0000","last_became_peered":"2023-09-30T00:17:51.590210+0000","last_unstale":"2023-09-30T00:17:51.962989+0000","last_undegraded":"2023-09-30T00:17:51.962989+0000","last_fullsized":"2023-09-30T00:17:51.962989+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:30:32.708671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.944019+0000","last_change":"2023-09-30T00:17:45.196688+0000","last_active":"2023-09-30T00:17:51.944019+0000","last_peered":"2023-09-30T00:17:51.944019+0000","last_clean":"2023-09-30T00:17:51.944019+0000","last_became_active":"2023-09-30T00:17:45.196361+0000","last_became_peered":"2023-09-30T00:17:45.196361+0000","last_unstale":"2023-09-30T00:17:51.944019+0000","last_undegraded":"2023-09-30T00:17:51.944019+0000","last_fullsized":"2023-09-30T00:17:51.944019+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.977735+0000","last_change":"2023-09-30T00:17:51.974386+0000","last_active":"2023-09-30T00:17:51.977735+0000","last_peered":"2023-09-30T00:17:51.977735+0000","last_clean":"2023-09-30T00:17:51.977735+0000","last_became_active":"2023-09-30T00:17:51.973776+0000","last_became_peered":"2023-09-30T00:17:51.973776+0000","last_unstale":"2023-09-30T00:17:51.977735+0000","last_undegraded":"2023-09-30T00:17:51.977735+0000","last_fullsized":"2023-09-30T00:17:51.977735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:25:21.282436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943386+0000","last_change":"2023-09-30T00:17:45.203067+0000","last_active":"2023-09-30T00:17:51.943386+0000","last_peered":"2023-09-30T00:17:51.943386+0000","last_clean":"2023-09-30T00:17:51.943386+0000","last_became_active":"2023-09-30T00:17:45.202823+0000","last_became_peered":"2023-09-30T00:17:45.202823+0000","last_unstale":"2023-09-30T00:17:51.943386+0000","last_undegraded":"2023-09-30T00:17:51.943386+0000","last_fullsized":"2023-09-30T00:17:51.943386+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:26:01.195302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-09-30T00:17:52.947337+0000","last_change":"2023-09-30T00:17:51.592843+0000","last_active":"2023-09-30T00:17:52.947337+0000","last_peered":"2023-09-30T00:17:52.947337+0000","last_clean":"2023-09-30T00:17:52.947337+0000","last_became_active":"2023-09-30T00:17:51.592601+0000","last_became_peered":"2023-09-30T00:17:51.592601+0000","last_unstale":"2023-09-30T00:17:52.947337+0000","last_undegraded":"2023-09-30T00:17:52.947337+0000","last_fullsized":"2023-09-30T00:17:52.947337+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:02:35.989080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.968789+0000","last_change":"2023-09-30T00:17:51.965286+0000","last_active":"2023-09-30T00:17:51.968789+0000","last_peered":"2023-09-30T00:17:51.968789+0000","last_clean":"2023-09-30T00:17:51.968789+0000","last_became_active":"2023-09-30T00:17:51.964869+0000","last_became_peered":"2023-09-30T00:17:51.964869+0000","last_unstale":"2023-09-30T00:17:51.968789+0000","last_undegraded":"2023-09-30T00:17:51.968789+0000","last_fullsized":"2023-09-30T00:17:51.968789+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:50:46.192135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943060+0000","last_change":"2023-09-30T00:17:45.195737+0000","last_active":"2023-09-30T00:17:51.943060+0000","last_peered":"2023-09-30T00:17:51.943060+0000","last_clean":"2023-09-30T00:17:51.943060+0000","last_became_active":"2023-09-30T00:17:45.195464+0000","last_became_peered":"2023-09-30T00:17:45.195464+0000","last_unstale":"2023-09-30T00:17:51.943060+0000","last_undegraded":"2023-09-30T00:17:51.943060+0000","last_fullsized":"2023-09-30T00:17:51.943060+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:24:48.730978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952759+0000","last_change":"2023-09-30T00:17:47.199132+0000","last_active":"2023-09-30T00:17:51.952759+0000","last_peered":"2023-09-30T00:17:51.952759+0000","last_clean":"2023-09-30T00:17:51.952759+0000","last_became_active":"2023-09-30T00:17:47.198255+0000","last_became_peered":"2023-09-30T00:17:47.198255+0000","last_unstale":"2023-09-30T00:17:51.952759+0000","last_undegraded":"2023-09-30T00:17:51.952759+0000","last_fullsized":"2023-09-30T00:17:51.952759+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:46.189460+0000","last_clean_scrub_stamp":"2023-09-30T00:17:46.189460+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:09.834201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943061+0000","last_change":"2023-09-30T00:17:51.575046+0000","last_active":"2023-09-30T00:17:51.943061+0000","last_peered":"2023-09-30T00:17:51.943061+0000","last_clean":"2023-09-30T00:17:51.943061+0000","last_became_active":"2023-09-30T00:17:51.574815+0000","last_became_peered":"2023-09-30T00:17:51.574815+0000","last_unstale":"2023-09-30T00:17:51.943061+0000","last_undegraded":"2023-09-30T00:17:51.943061+0000","last_fullsized":"2023-09-30T00:17:51.943061+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:43:57.892326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973520+0000","last_change":"2023-09-30T00:17:51.969276+0000","last_active":"2023-09-30T00:17:51.973520+0000","last_peered":"2023-09-30T00:17:51.973520+0000","last_clean":"2023-09-30T00:17:51.973520+0000","last_became_active":"2023-09-30T00:17:51.966513+0000","last_became_peered":"2023-09-30T00:17:51.966513+0000","last_unstale":"2023-09-30T00:17:51.973520+0000","last_undegraded":"2023-09-30T00:17:51.973520+0000","last_fullsized":"2023-09-30T00:17:51.973520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:41:55.301919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.946321+0000","last_change":"2023-09-30T00:17:45.197230+0000","last_active":"2023-09-30T00:17:51.946321+0000","last_peered":"2023-09-30T00:17:51.946321+0000","last_clean":"2023-09-30T00:17:51.946321+0000","last_became_active":"2023-09-30T00:17:45.196528+0000","last_became_peered":"2023-09-30T00:17:45.196528+0000","last_unstale":"2023-09-30T00:17:51.946321+0000","last_undegraded":"2023-09-30T00:17:51.946321+0000","last_fullsized":"2023-09-30T00:17:51.946321+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941919+0000","last_change":"2023-09-30T00:17:51.575520+0000","last_active":"2023-09-30T00:17:51.941919+0000","last_peered":"2023-09-30T00:17:51.941919+0000","last_clean":"2023-09-30T00:17:51.941919+0000","last_became_active":"2023-09-30T00:17:51.575306+0000","last_became_peered":"2023-09-30T00:17:51.575306+0000","last_unstale":"2023-09-30T00:17:51.941919+0000","last_undegraded":"2023-09-30T00:17:51.941919+0000","last_fullsized":"2023-09-30T00:17:51.941919+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:31:32.127076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.959411+0000","last_change":"2023-09-30T00:17:51.958852+0000","last_active":"2023-09-30T00:17:51.959411+0000","last_peered":"2023-09-30T00:17:51.959411+0000","last_clean":"2023-09-30T00:17:51.959411+0000","last_became_active":"2023-09-30T00:17:51.958570+0000","last_became_peered":"2023-09-30T00:17:51.958570+0000","last_unstale":"2023-09-30T00:17:51.959411+0000","last_undegraded":"2023-09-30T00:17:51.959411+0000","last_fullsized":"2023-09-30T00:17:51.959411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:34:27.611487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.940896+0000","last_change":"2023-09-30T00:17:45.196542+0000","last_active":"2023-09-30T00:17:51.940896+0000","last_peered":"2023-09-30T00:17:51.940896+0000","last_clean":"2023-09-30T00:17:51.940896+0000","last_became_active":"2023-09-30T00:17:45.196123+0000","last_became_peered":"2023-09-30T00:17:45.196123+0000","last_unstale":"2023-09-30T00:17:51.940896+0000","last_undegraded":"2023-09-30T00:17:51.940896+0000","last_fullsized":"2023-09-30T00:17:51.940896+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952371+0000","last_change":"2023-09-30T00:17:51.591518+0000","last_active":"2023-09-30T00:17:51.952371+0000","last_peered":"2023-09-30T00:17:51.952371+0000","last_clean":"2023-09-30T00:17:51.952371+0000","last_became_active":"2023-09-30T00:17:51.591253+0000","last_became_peered":"2023-09-30T00:17:51.591253+0000","last_unstale":"2023-09-30T00:17:51.952371+0000","last_undegraded":"2023-09-30T00:17:51.952371+0000","last_fullsized":"2023-09-30T00:17:51.952371+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:52:46.330854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.980323+0000","last_change":"2023-09-30T00:17:51.976800+0000","last_active":"2023-09-30T00:17:51.980323+0000","last_peered":"2023-09-30T00:17:51.980323+0000","last_clean":"2023-09-30T00:17:51.980323+0000","last_became_active":"2023-09-30T00:17:51.976602+0000","last_became_peered":"2023-09-30T00:17:51.976602+0000","last_unstale":"2023-09-30T00:17:51.980323+0000","last_undegraded":"2023-09-30T00:17:51.980323+0000","last_fullsized":"2023-09-30T00:17:51.980323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:53:26.550299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.952216+0000","last_change":"2023-09-30T00:17:45.207123+0000","last_active":"2023-09-30T00:17:51.952216+0000","last_peered":"2023-09-30T00:17:51.952216+0000","last_clean":"2023-09-30T00:17:51.952216+0000","last_became_active":"2023-09-30T00:17:45.206750+0000","last_became_peered":"2023-09-30T00:17:45.206750+0000","last_unstale":"2023-09-30T00:17:51.952216+0000","last_undegraded":"2023-09-30T00:17:51.952216+0000","last_fullsized":"2023-09-30T00:17:51.952216+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:26:47.049739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.945122+0000","last_change":"2023-09-30T00:17:51.585515+0000","last_active":"2023-09-30T00:17:51.945122+0000","last_peered":"2023-09-30T00:17:51.945122+0000","last_clean":"2023-09-30T00:17:51.945122+0000","last_became_active":"2023-09-30T00:17:51.585262+0000","last_became_peered":"2023-09-30T00:17:51.585262+0000","last_unstale":"2023-09-30T00:17:51.945122+0000","last_undegraded":"2023-09-30T00:17:51.945122+0000","last_fullsized":"2023-09-30T00:17:51.945122+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:04:04.914282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960357+0000","last_change":"2023-09-30T00:17:51.959726+0000","last_active":"2023-09-30T00:17:51.960357+0000","last_peered":"2023-09-30T00:17:51.960357+0000","last_clean":"2023-09-30T00:17:51.960357+0000","last_became_active":"2023-09-30T00:17:51.959328+0000","last_became_peered":"2023-09-30T00:17:51.959328+0000","last_unstale":"2023-09-30T00:17:51.960357+0000","last_undegraded":"2023-09-30T00:17:51.960357+0000","last_fullsized":"2023-09-30T00:17:51.960357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:37:08.356297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.954829+0000","last_change":"2023-09-30T00:17:45.206971+0000","last_active":"2023-09-30T00:17:51.954829+0000","last_peered":"2023-09-30T00:17:51.954829+0000","last_clean":"2023-09-30T00:17:51.954829+0000","last_became_active":"2023-09-30T00:17:45.206693+0000","last_became_peered":"2023-09-30T00:17:45.206693+0000","last_unstale":"2023-09-30T00:17:51.954829+0000","last_undegraded":"2023-09-30T00:17:51.954829+0000","last_fullsized":"2023-09-30T00:17:51.954829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:26:42.423718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.943205+0000","last_change":"2023-09-30T00:17:51.575506+0000","last_active":"2023-09-30T00:17:51.943205+0000","last_peered":"2023-09-30T00:17:51.943205+0000","last_clean":"2023-09-30T00:17:51.943205+0000","last_became_active":"2023-09-30T00:17:51.574806+0000","last_became_peered":"2023-09-30T00:17:51.574806+0000","last_unstale":"2023-09-30T00:17:51.943205+0000","last_undegraded":"2023-09-30T00:17:51.943205+0000","last_fullsized":"2023-09-30T00:17:51.943205+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:51.816991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975408+0000","last_change":"2023-09-30T00:17:51.967785+0000","last_active":"2023-09-30T00:17:51.975408+0000","last_peered":"2023-09-30T00:17:51.975408+0000","last_clean":"2023-09-30T00:17:51.975408+0000","last_became_active":"2023-09-30T00:17:51.967419+0000","last_became_peered":"2023-09-30T00:17:51.967419+0000","last_unstale":"2023-09-30T00:17:51.975408+0000","last_undegraded":"2023-09-30T00:17:51.975408+0000","last_fullsized":"2023-09-30T00:17:51.975408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.941618+0000","last_change":"2023-09-30T00:17:45.203202+0000","last_active":"2023-09-30T00:17:51.941618+0000","last_peered":"2023-09-30T00:17:51.941618+0000","last_clean":"2023-09-30T00:17:51.941618+0000","last_became_active":"2023-09-30T00:17:45.202879+0000","last_became_peered":"2023-09-30T00:17:45.202879+0000","last_unstale":"2023-09-30T00:17:51.941618+0000","last_undegraded":"2023-09-30T00:17:51.941618+0000","last_fullsized":"2023-09-30T00:17:51.941618+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:44.183678+0000","last_clean_scrub_stamp":"2023-09-30T00:17:44.183678+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:50:42.379943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.944043+0000","last_change":"2023-09-30T00:17:51.580771+0000","last_active":"2023-09-30T00:17:51.944043+0000","last_peered":"2023-09-30T00:17:51.944043+0000","last_clean":"2023-09-30T00:17:51.944043+0000","last_became_active":"2023-09-30T00:17:51.580495+0000","last_became_peered":"2023-09-30T00:17:51.580495+0000","last_unstale":"2023-09-30T00:17:51.944043+0000","last_undegraded":"2023-09-30T00:17:51.944043+0000","last_fullsized":"2023-09-30T00:17:51.944043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.971832+0000","last_change":"2023-09-30T00:17:51.957745+0000","last_active":"2023-09-30T00:17:51.971832+0000","last_peered":"2023-09-30T00:17:51.971832+0000","last_clean":"2023-09-30T00:17:51.971832+0000","last_became_active":"2023-09-30T00:17:51.955331+0000","last_became_peered":"2023-09-30T00:17:51.955331+0000","last_unstale":"2023-09-30T00:17:51.971832+0000","last_undegraded":"2023-09-30T00:17:51.971832+0000","last_fullsized":"2023-09-30T00:17:51.971832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:45.393660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.973877+0000","last_change":"2023-09-30T00:17:51.596021+0000","last_active":"2023-09-30T00:17:51.973877+0000","last_peered":"2023-09-30T00:17:51.973877+0000","last_clean":"2023-09-30T00:17:51.973877+0000","last_became_active":"2023-09-30T00:17:51.595745+0000","last_became_peered":"2023-09-30T00:17:51.595745+0000","last_unstale":"2023-09-30T00:17:51.973877+0000","last_undegraded":"2023-09-30T00:17:51.973877+0000","last_fullsized":"2023-09-30T00:17:51.973877+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:47:44.390191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960330+0000","last_change":"2023-09-30T00:17:51.959564+0000","last_active":"2023-09-30T00:17:51.960330+0000","last_peered":"2023-09-30T00:17:51.960330+0000","last_clean":"2023-09-30T00:17:51.960330+0000","last_became_active":"2023-09-30T00:17:51.959278+0000","last_became_peered":"2023-09-30T00:17:51.959278+0000","last_unstale":"2023-09-30T00:17:51.960330+0000","last_undegraded":"2023-09-30T00:17:51.960330+0000","last_fullsized":"2023-09-30T00:17:51.960330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:55.932612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.974567+0000","last_change":"2023-09-30T00:17:51.587276+0000","last_active":"2023-09-30T00:17:51.974567+0000","last_peered":"2023-09-30T00:17:51.974567+0000","last_clean":"2023-09-30T00:17:51.974567+0000","last_became_active":"2023-09-30T00:17:51.586984+0000","last_became_peered":"2023-09-30T00:17:51.586984+0000","last_unstale":"2023-09-30T00:17:51.974567+0000","last_undegraded":"2023-09-30T00:17:51.974567+0000","last_fullsized":"2023-09-30T00:17:51.974567+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:35:05.714598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960802+0000","last_change":"2023-09-30T00:17:51.953787+0000","last_active":"2023-09-30T00:17:51.960802+0000","last_peered":"2023-09-30T00:17:51.960802+0000","last_clean":"2023-09-30T00:17:51.960802+0000","last_became_active":"2023-09-30T00:17:51.951805+0000","last_became_peered":"2023-09-30T00:17:51.951805+0000","last_unstale":"2023-09-30T00:17:51.960802+0000","last_undegraded":"2023-09-30T00:17:51.960802+0000","last_fullsized":"2023-09-30T00:17:51.960802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:57:04.290243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"18'2","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.965610+0000","last_change":"2023-09-30T00:17:51.593225+0000","last_active":"2023-09-30T00:17:51.965610+0000","last_peered":"2023-09-30T00:17:51.965610+0000","last_clean":"2023-09-30T00:17:51.965610+0000","last_became_active":"2023-09-30T00:17:51.592913+0000","last_became_peered":"2023-09-30T00:17:51.592913+0000","last_unstale":"2023-09-30T00:17:51.965610+0000","last_undegraded":"2023-09-30T00:17:51.965610+0000","last_fullsized":"2023-09-30T00:17:51.965610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:37:56.776343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.961497+0000","last_change":"2023-09-30T00:17:51.955058+0000","last_active":"2023-09-30T00:17:51.961497+0000","last_peered":"2023-09-30T00:17:51.961497+0000","last_clean":"2023-09-30T00:17:51.961497+0000","last_became_active":"2023-09-30T00:17:51.954787+0000","last_became_peered":"2023-09-30T00:17:51.954787+0000","last_unstale":"2023-09-30T00:17:51.961497+0000","last_undegraded":"2023-09-30T00:17:51.961497+0000","last_fullsized":"2023-09-30T00:17:51.961497+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:49:30.148218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"18'1","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960867+0000","last_change":"2023-09-30T00:17:51.586229+0000","last_active":"2023-09-30T00:17:51.960867+0000","last_peered":"2023-09-30T00:17:51.960867+0000","last_clean":"2023-09-30T00:17:51.960867+0000","last_became_active":"2023-09-30T00:17:51.586027+0000","last_became_peered":"2023-09-30T00:17:51.586027+0000","last_unstale":"2023-09-30T00:17:51.960867+0000","last_undegraded":"2023-09-30T00:17:51.960867+0000","last_fullsized":"2023-09-30T00:17:51.960867+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:48:13.817514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.962963+0000","last_change":"2023-09-30T00:17:51.959628+0000","last_active":"2023-09-30T00:17:51.962963+0000","last_peered":"2023-09-30T00:17:51.962963+0000","last_clean":"2023-09-30T00:17:51.962963+0000","last_became_active":"2023-09-30T00:17:51.958974+0000","last_became_peered":"2023-09-30T00:17:51.958974+0000","last_unstale":"2023-09-30T00:17:51.962963+0000","last_undegraded":"2023-09-30T00:17:51.962963+0000","last_fullsized":"2023-09-30T00:17:51.962963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:07:28.604134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"18'4","reported_seq":14,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960480+0000","last_change":"2023-09-30T00:17:51.590387+0000","last_active":"2023-09-30T00:17:51.960480+0000","last_peered":"2023-09-30T00:17:51.960480+0000","last_clean":"2023-09-30T00:17:51.960480+0000","last_became_active":"2023-09-30T00:17:51.585479+0000","last_became_peered":"2023-09-30T00:17:51.585479+0000","last_unstale":"2023-09-30T00:17:51.960480+0000","last_undegraded":"2023-09-30T00:17:51.960480+0000","last_fullsized":"2023-09-30T00:17:51.960480+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:50.560693+0000","last_clean_scrub_stamp":"2023-09-30T00:17:50.560693+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:48.868216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.959772+0000","last_change":"2023-09-30T00:17:51.949418+0000","last_active":"2023-09-30T00:17:51.959772+0000","last_peered":"2023-09-30T00:17:51.959772+0000","last_clean":"2023-09-30T00:17:51.959772+0000","last_became_active":"2023-09-30T00:17:51.949101+0000","last_became_peered":"2023-09-30T00:17:51.949101+0000","last_unstale":"2023-09-30T00:17:51.959772+0000","last_undegraded":"2023-09-30T00:17:51.959772+0000","last_fullsized":"2023-09-30T00:17:51.959772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:07:23.811447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.975274+0000","last_change":"2023-09-30T00:17:51.964155+0000","last_active":"2023-09-30T00:17:51.975274+0000","last_peered":"2023-09-30T00:17:51.975274+0000","last_clean":"2023-09-30T00:17:51.975274+0000","last_became_active":"2023-09-30T00:17:51.958334+0000","last_became_peered":"2023-09-30T00:17:51.958334+0000","last_unstale":"2023-09-30T00:17:51.975274+0000","last_undegraded":"2023-09-30T00:17:51.975274+0000","last_fullsized":"2023-09-30T00:17:51.975274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:10:59.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.959645+0000","last_change":"2023-09-30T00:17:51.959066+0000","last_active":"2023-09-30T00:17:51.959645+0000","last_peered":"2023-09-30T00:17:51.959645+0000","last_clean":"2023-09-30T00:17:51.959645+0000","last_became_active":"2023-09-30T00:17:51.958741+0000","last_became_peered":"2023-09-30T00:17:51.958741+0000","last_unstale":"2023-09-30T00:17:51.959645+0000","last_undegraded":"2023-09-30T00:17:51.959645+0000","last_fullsized":"2023-09-30T00:17:51.959645+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:54:35.602823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.976104+0000","last_change":"2023-09-30T00:17:51.970791+0000","last_active":"2023-09-30T00:17:51.976104+0000","last_peered":"2023-09-30T00:17:51.976104+0000","last_clean":"2023-09-30T00:17:51.976104+0000","last_became_active":"2023-09-30T00:17:51.964142+0000","last_became_peered":"2023-09-30T00:17:51.964142+0000","last_unstale":"2023-09-30T00:17:51.976104+0000","last_undegraded":"2023-09-30T00:17:51.976104+0000","last_fullsized":"2023-09-30T00:17:51.976104+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:35:11.495087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.967656+0000","last_change":"2023-09-30T00:17:51.967167+0000","last_active":"2023-09-30T00:17:51.967656+0000","last_peered":"2023-09-30T00:17:51.967656+0000","last_clean":"2023-09-30T00:17:51.967656+0000","last_became_active":"2023-09-30T00:17:51.966937+0000","last_became_peered":"2023-09-30T00:17:51.966937+0000","last_unstale":"2023-09-30T00:17:51.967656+0000","last_undegraded":"2023-09-30T00:17:51.967656+0000","last_fullsized":"2023-09-30T00:17:51.967656+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:32:11.138421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-30T00:17:51.960344+0000","last_change":"2023-09-30T00:17:51.959728+0000","last_active":"2023-09-30T00:17:51.960344+0000","last_peered":"2023-09-30T00:17:51.960344+0000","last_clean":"2023-09-30T00:17:51.960344+0000","last_became_active":"2023-09-30T00:17:51.959512+0000","last_became_peered":"2023-09-30T00:17:51.959512+0000","last_unstale":"2023-09-30T00:17:51.960344+0000","last_undegraded":"2023-09-30T00:17:51.960344+0000","last_fullsized":"2023-09-30T00:17:51.960344+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:17:51.564162+0000","last_clean_scrub_stamp":"2023-09-30T00:17:51.564162+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:05:15.182292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672968,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9232,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362608,"statfs":{"total":96636764160,"available":96627310592,"internally_reserved":0,"allocated":794624,"data_stored":651535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]},{"osd":2,"up_from":10,"seq":42949672968,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8072,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363768,"statfs":{"total":96636764160,"available":96628498432,"internally_reserved":0,"allocated":196608,"data_stored":60391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9084,"kb_used_data":756,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94362756,"statfs":{"total":96636764160,"available":96627462144,"internally_reserved":0,"allocated":774144,"data_stored":650359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":1,"up_from":10,"seq":42949672968,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8064,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363776,"statfs":{"total":96636764160,"available":96628506624,"internally_reserved":0,"allocated":188416,"data_stored":60083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:18:06.031 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:18:06.031 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T00:18:06.031 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T00:18:06.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.180+0000 7f7e483bf700 1 Processor -- start 2023-09-30T00:18:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.180+0000 7f7e483bf700 1 -- start start 2023-09-30T00:18:06.183 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.180+0000 7f7e483bf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 0x7f7e400a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:06.183 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.180+0000 7f7e483bf700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7e400b7160 0x7f7e400a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:06.183 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.180+0000 7f7e483bf700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e400a1310 con 0x7f7e4013fd70 2023-09-30T00:18:06.183 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.180+0000 7f7e483bf700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e40071020 con 0x7f7e400b7160 2023-09-30T00:18:06.183 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.180+0000 7f7e483bf700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e400b76a0 con 0x7f7e40070c50 2023-09-30T00:18:06.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.180+0000 7f7e4615b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 0x7f7e400a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:06.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e4695c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7e4013fd70 0x7f7e400b4dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46428/0 (socket says 172.21.15.94:46428) 2023-09-30T00:18:06.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e4695c700 1 -- 172.21.15.94:0/4125137224 learned_addr learned my addr 172.21.15.94:0/4125137224 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:06.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e4595a700 1 --2- 172.21.15.94:0/4125137224 >> v2:172.21.15.121:3300/0 conn(0x7f7e400b7160 0x7f7e400a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:06.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e4615b700 1 -- 172.21.15.94:0/4125137224 >> v2:172.21.15.121:3300/0 conn(0x7f7e400b7160 msgr2=0x7f7e400a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:06.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e45159700 1 -- 172.21.15.94:0/4125137224 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975267622 0 0) 0x7f7e38001490 con 0x7f7e4013fd70 2023-09-30T00:18:06.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e4615b700 1 --2- 172.21.15.94:0/4125137224 >> v2:172.21.15.121:3300/0 conn(0x7f7e400b7160 0x7f7e400a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e4615b700 1 -- 172.21.15.94:0/4125137224 >> v1:172.21.15.94:6789/0 conn(0x7f7e4013fd70 legacy=0x7f7e400b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:06.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e4615b700 1 -- 172.21.15.94:0/4125137224 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e400b7940 con 0x7f7e40070c50 2023-09-30T00:18:06.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e4595a700 1 --2- 172.21.15.94:0/4125137224 >> v2:172.21.15.121:3300/0 conn(0x7f7e400b7160 0x7f7e400a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:18:06.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e4615b700 1 --2- 172.21.15.94:0/4125137224 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 0x7f7e400a0dd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0098e0 tx=0x7f7e3c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df84724ce2c9e705 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:06.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e45159700 1 -- 172.21.15.94:0/4125137224 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c009e80 con 0x7f7e40070c50 2023-09-30T00:18:06.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e45159700 1 -- 172.21.15.94:0/4125137224 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c00b8f0 con 0x7f7e40070c50 2023-09-30T00:18:06.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.181+0000 7f7e45159700 1 -- 172.21.15.94:0/4125137224 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c00bc70 con 0x7f7e40070c50 2023-09-30T00:18:06.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.182+0000 7f7e483bf700 1 -- 172.21.15.94:0/4125137224 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 msgr2=0x7f7e400a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:06.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.182+0000 7f7e483bf700 1 --2- 172.21.15.94:0/4125137224 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 0x7f7e400a0dd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0098e0 tx=0x7f7e3c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.182+0000 7f7e483bf700 1 -- 172.21.15.94:0/4125137224 shutdown_connections 2023-09-30T00:18:06.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.182+0000 7f7e483bf700 1 --2- 172.21.15.94:0/4125137224 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 0x7f7e400a0dd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.182+0000 7f7e483bf700 1 --2- 172.21.15.94:0/4125137224 >> v2:172.21.15.121:3300/0 conn(0x7f7e400b7160 0x7f7e400a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.182+0000 7f7e483bf700 1 -- 172.21.15.94:0/4125137224 >> 172.21.15.94:0/4125137224 conn(0x7f7e4006a090 msgr2=0x7f7e400702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:06.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.182+0000 7f7e483bf700 1 -- 172.21.15.94:0/4125137224 shutdown_connections 2023-09-30T00:18:06.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.182+0000 7f7e483bf700 1 -- 172.21.15.94:0/4125137224 wait complete. 2023-09-30T00:18:06.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e483bf700 1 Processor -- start 2023-09-30T00:18:06.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e483bf700 1 -- start start 2023-09-30T00:18:06.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e483bf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 0x7f7e40150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:06.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e483bf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e400b7160 0x7f7e4013bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:06.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e483bf700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e4013fd70 0x7f7e400abb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:06.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e483bf700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e400adc50 con 0x7f7e4013fd70 2023-09-30T00:18:06.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e483bf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e400aca30 con 0x7f7e400b7160 2023-09-30T00:18:06.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e483bf700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e40151650 con 0x7f7e40070c50 2023-09-30T00:18:06.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e4615b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 0x7f7e40150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:06.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e4615b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 0x7f7e40150480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:37352/0 (socket says 172.21.15.94:37352) 2023-09-30T00:18:06.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e4615b700 1 -- 172.21.15.94:0/3533718138 learned_addr learned my addr 172.21.15.94:0/3533718138 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:06.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e4695c700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e4013fd70 0x7f7e400abb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:06.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e4595a700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e400b7160 0x7f7e4013bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:06.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e4615b700 1 -- 172.21.15.94:0/3533718138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e400b7160 msgr2=0x7f7e4013bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:06.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e4615b700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e400b7160 0x7f7e4013bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e4615b700 1 -- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e4013fd70 msgr2=0x7f7e400abb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:06.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e4615b700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e4013fd70 0x7f7e400abb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.183+0000 7f7e4615b700 1 -- 172.21.15.94:0/3533718138 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c009420 con 0x7f7e40070c50 2023-09-30T00:18:06.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.184+0000 7f7e4595a700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e400b7160 0x7f7e4013bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:06.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.184+0000 7f7e4695c700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e4013fd70 0x7f7e400abb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.184+0000 7f7e4615b700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 0x7f7e40150480 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c006ef0 tx=0x7f7e3c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.184+0000 7f7e367fc700 1 -- 172.21.15.94:0/3533718138 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c00bb30 con 0x7f7e40070c50 2023-09-30T00:18:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.184+0000 7f7e367fc700 1 -- 172.21.15.94:0/3533718138 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c025420 con 0x7f7e40070c50 2023-09-30T00:18:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.184+0000 7f7e483bf700 1 -- 172.21.15.94:0/3533718138 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e40150d40 con 0x7f7e40070c50 2023-09-30T00:18:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.184+0000 7f7e367fc700 1 -- 172.21.15.94:0/3533718138 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c0257e0 con 0x7f7e40070c50 2023-09-30T00:18:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.184+0000 7f7e483bf700 1 -- 172.21.15.94:0/3533718138 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e400acd10 con 0x7f7e40070c50 2023-09-30T00:18:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.185+0000 7f7e483bf700 1 -- 172.21.15.94:0/3533718138 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e400a83e0 con 0x7f7e40070c50 2023-09-30T00:18:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.186+0000 7f7e367fc700 1 -- 172.21.15.94:0/3533718138 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f7e3c023070 con 0x7f7e40070c50 2023-09-30T00:18:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.186+0000 7f7e367fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.186+0000 7f7e367fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.186+0000 7f7e367fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:06.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.186+0000 7f7e367fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:06.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.186+0000 7f7e367fc700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7e24098780 0x7f7e2409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:06.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.186+0000 7f7e4595a700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7e24098780 0x7f7e2409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:06.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.186+0000 7f7e367fc700 1 -- 172.21.15.94:0/3533718138 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f7e3c0c1c30 con 0x7f7e40070c50 2023-09-30T00:18:06.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.186+0000 7f7e4595a700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7e24098780 0x7f7e2409ac30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7e4006a5a0 tx=0x7f7e30006f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:06.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.190+0000 7f7e367fc700 1 -- 172.21.15.94:0/3533718138 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e3c08f2e0 con 0x7f7e40070c50 2023-09-30T00:18:06.417 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.417+0000 7f7e483bf700 1 -- 172.21.15.94:0/3533718138 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7e40071630 con 0x7f7e40070c50 2023-09-30T00:18:06.418 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.417+0000 7f7e367fc700 1 -- 172.21.15.94:0/3533718138 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7e3c02a100 con 0x7f7e40070c50 2023-09-30T00:18:06.418 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:06.418 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 10 mgrc shutdown 2023-09-30T00:18:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 1 -- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7e24098780 msgr2=0x7f7e2409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7e24098780 0x7f7e2409ac30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7e4006a5a0 tx=0x7f7e30006f90 comp rx=0 tx=0).stop 2023-09-30T00:18:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 1 -- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 msgr2=0x7f7e40150480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 0x7f7e40150480 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c006ef0 tx=0x7f7e3c017000 comp rx=0 tx=0).stop 2023-09-30T00:18:06.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 1 -- 172.21.15.94:0/3533718138 shutdown_connections 2023-09-30T00:18:06.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7e24098780 0x7f7e2409ac30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e40070c50 0x7f7e40150480 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e4013fd70 0x7f7e400abb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 1 --2- 172.21.15.94:0/3533718138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e400b7160 0x7f7e4013bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:06.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 1 -- 172.21.15.94:0/3533718138 >> 172.21.15.94:0/3533718138 conn(0x7f7e4006a090 msgr2=0x7f7e4013cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:06.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 1 -- 172.21.15.94:0/3533718138 shutdown_connections 2023-09-30T00:18:06.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:06.419+0000 7f7e483bf700 1 -- 172.21.15.94:0/3533718138 wait complete. 2023-09-30T00:18:08.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:18:08.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:18:09.435 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe53a49a700 1 Processor -- start 2023-09-30T00:18:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe53a49a700 1 -- start start 2023-09-30T00:18:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe53a49a700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe5340b7160 0x7fe5340a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:09.562 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe53a49a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe53413fd70 0x7fe5341401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe53a49a700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5340a1310 con 0x7fe534070c50 2023-09-30T00:18:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe53a49a700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe534071020 con 0x7fe5340b7160 2023-09-30T00:18:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe53a49a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5340b76a0 con 0x7fe53413fd70 2023-09-30T00:18:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe538a37700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe53413fd70 0x7fe5341401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe5337fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe5340b7160 0x7fe5340a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe538a37700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe53413fd70 0x7fe5341401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:37364/0 (socket says 172.21.15.94:37364) 2023-09-30T00:18:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe5337fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe5340b7160 0x7fe5340a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36286/0 (socket says 172.21.15.94:36286) 2023-09-30T00:18:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe538a37700 1 -- 172.21.15.94:0/1382274844 learned_addr learned my addr 172.21.15.94:0/1382274844 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.560+0000 7fe533fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe534070c50 0x7fe5340a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46442/0 (socket says 172.21.15.94:46442) 2023-09-30T00:18:09.563 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.561+0000 7fe538a37700 1 -- 172.21.15.94:0/1382274844 >> v2:172.21.15.94:3300/0 conn(0x7fe5340b7160 msgr2=0x7fe5340a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.561+0000 7fe538a37700 1 --2- 172.21.15.94:0/1382274844 >> v2:172.21.15.94:3300/0 conn(0x7fe5340b7160 0x7fe5340a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.561+0000 7fe538a37700 1 -- 172.21.15.94:0/1382274844 >> v1:172.21.15.94:6789/0 conn(0x7fe534070c50 legacy=0x7fe5340a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.561+0000 7fe538a37700 1 -- 172.21.15.94:0/1382274844 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5340b7940 con 0x7fe53413fd70 2023-09-30T00:18:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.561+0000 7fe5337fe700 1 --2- 172.21.15.94:0/1382274844 >> v2:172.21.15.94:3300/0 conn(0x7fe5340b7160 0x7fe5340a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.561+0000 7fe532ffd700 1 -- 172.21.15.94:0/1382274844 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877125869 0 0) 0x7fe524001110 con 0x7fe534070c50 2023-09-30T00:18:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.561+0000 7fe538a37700 1 --2- 172.21.15.94:0/1382274844 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe53413fd70 0x7fe5341401d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009f20 tx=0x7fe52c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78f9816aa55c369 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.561+0000 7fe532ffd700 1 -- 172.21.15.94:0/1382274844 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c00e070 con 0x7fe53413fd70 2023-09-30T00:18:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.561+0000 7fe532ffd700 1 -- 172.21.15.94:0/1382274844 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52c00ba60 con 0x7fe53413fd70 2023-09-30T00:18:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.561+0000 7fe532ffd700 1 -- 172.21.15.94:0/1382274844 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c0092e0 con 0x7fe53413fd70 2023-09-30T00:18:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.561+0000 7fe53a49a700 1 -- 172.21.15.94:0/1382274844 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe53413fd70 msgr2=0x7fe5341401d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:09.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.562+0000 7fe53a49a700 1 --2- 172.21.15.94:0/1382274844 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe53413fd70 0x7fe5341401d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009f20 tx=0x7fe52c009840 comp rx=0 tx=0).stop 2023-09-30T00:18:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.562+0000 7fe53a49a700 1 -- 172.21.15.94:0/1382274844 shutdown_connections 2023-09-30T00:18:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.562+0000 7fe53a49a700 1 --2- 172.21.15.94:0/1382274844 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe53413fd70 0x7fe5341401d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.562+0000 7fe53a49a700 1 --2- 172.21.15.94:0/1382274844 >> v2:172.21.15.94:3300/0 conn(0x7fe5340b7160 0x7fe5340a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.562+0000 7fe53a49a700 1 -- 172.21.15.94:0/1382274844 >> 172.21.15.94:0/1382274844 conn(0x7fe53406a090 msgr2=0x7fe5340702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.562+0000 7fe53a49a700 1 -- 172.21.15.94:0/1382274844 shutdown_connections 2023-09-30T00:18:09.565 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.562+0000 7fe53a49a700 1 -- 172.21.15.94:0/1382274844 wait complete. 2023-09-30T00:18:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.562+0000 7fe53a49a700 1 Processor -- start 2023-09-30T00:18:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.563+0000 7fe53a49a700 1 -- start start 2023-09-30T00:18:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.563+0000 7fe53a49a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe534070c50 0x7fe5340a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.563+0000 7fe53a49a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5340b7160 0x7fe534153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.563+0000 7fe53a49a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe53413fd70 0x7fe5341543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.563+0000 7fe53a49a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe534154b20 con 0x7fe53413fd70 2023-09-30T00:18:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.563+0000 7fe53a49a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe534154910 con 0x7fe534070c50 2023-09-30T00:18:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.563+0000 7fe53a49a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53413c740 con 0x7fe5340b7160 2023-09-30T00:18:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.563+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5340b7160 0x7fe534153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.563+0000 7fe538a37700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe53413fd70 0x7fe5341543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.563+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5340b7160 0x7fe534153e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:37380/0 (socket says 172.21.15.94:37380) 2023-09-30T00:18:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.563+0000 7fe5337fe700 1 -- 172.21.15.94:0/3553107902 learned_addr learned my addr 172.21.15.94:0/3553107902 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe533fff700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe534070c50 0x7fe5340a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe5337fe700 1 -- 172.21.15.94:0/3553107902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe534070c50 msgr2=0x7fe5340a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe5337fe700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe534070c50 0x7fe5340a67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe5337fe700 1 -- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe53413fd70 msgr2=0x7fe5341543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe5337fe700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe53413fd70 0x7fe5341543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe5337fe700 1 -- 172.21.15.94:0/3553107902 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe528009420 con 0x7fe5340b7160 2023-09-30T00:18:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe533fff700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe534070c50 0x7fe5340a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:18:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe538a37700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe53413fd70 0x7fe5341543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe5337fe700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5340b7160 0x7fe534153e90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe52800ec80 tx=0x7fe5280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe51bfff700 1 -- 172.21.15.94:0/3553107902 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52800c980 con 0x7fe5340b7160 2023-09-30T00:18:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe51bfff700 1 -- 172.21.15.94:0/3553107902 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52800cae0 con 0x7fe5340b7160 2023-09-30T00:18:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe53a49a700 1 -- 172.21.15.94:0/3553107902 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c009ae0 con 0x7fe5340b7160 2023-09-30T00:18:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe51bfff700 1 -- 172.21.15.94:0/3553107902 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52800ce60 con 0x7fe5340b7160 2023-09-30T00:18:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.564+0000 7fe53a49a700 1 -- 172.21.15.94:0/3553107902 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5340a5880 con 0x7fe5340b7160 2023-09-30T00:18:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.565+0000 7fe53a49a700 1 -- 172.21.15.94:0/3553107902 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe534153bb0 con 0x7fe5340b7160 2023-09-30T00:18:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.566+0000 7fe51bfff700 1 -- 172.21.15.94:0/3553107902 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe528013680 con 0x7fe5340b7160 2023-09-30T00:18:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.566+0000 7fe51bfff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.566+0000 7fe51bfff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.566+0000 7fe51bfff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.566+0000 7fe51bfff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.566+0000 7fe51bfff700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe520098780 0x7fe52009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.566+0000 7fe51bfff700 1 -- 172.21.15.94:0/3553107902 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fe5280c0880 con 0x7fe5340b7160 2023-09-30T00:18:09.570 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.566+0000 7fe533fff700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe520098780 0x7fe52009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:09.571 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.567+0000 7fe533fff700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe520098780 0x7fe52009ac30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe5240018c0 tx=0x7fe524009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:09.571 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.571+0000 7fe51bfff700 1 -- 172.21.15.94:0/3553107902 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe534153bb0 con 0x7fe5340b7160 2023-09-30T00:18:09.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.792+0000 7fe53a49a700 1 -- 172.21.15.94:0/3553107902 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe534153bb0 con 0x7fe5340b7160 2023-09-30T00:18:09.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.793+0000 7fe51bfff700 1 -- 172.21.15.94:0/3553107902 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe528021100 con 0x7fe5340b7160 2023-09-30T00:18:09.793 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:09.794 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:09.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 10 mgrc shutdown 2023-09-30T00:18:09.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 1 -- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe520098780 msgr2=0x7fe52009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:09.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe520098780 0x7fe52009ac30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe5240018c0 tx=0x7fe524009040 comp rx=0 tx=0).stop 2023-09-30T00:18:09.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 1 -- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5340b7160 msgr2=0x7fe534153e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:09.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5340b7160 0x7fe534153e90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe52800ec80 tx=0x7fe5280091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:09.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 1 -- 172.21.15.94:0/3553107902 shutdown_connections 2023-09-30T00:18:09.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe520098780 0x7fe52009ac30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:09.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe53413fd70 0x7fe5341543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:09.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe534070c50 0x7fe5340a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:09.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 1 --2- 172.21.15.94:0/3553107902 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5340b7160 0x7fe534153e90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:09.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 1 -- 172.21.15.94:0/3553107902 >> 172.21.15.94:0/3553107902 conn(0x7fe53406a090 msgr2=0x7fe5340685c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:09.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 1 -- 172.21.15.94:0/3553107902 shutdown_connections 2023-09-30T00:18:09.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:09.795+0000 7fe53a49a700 1 -- 172.21.15.94:0/3553107902 wait complete. 2023-09-30T00:18:12.807 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.928+0000 7f5395f5b700 1 Processor -- start 2023-09-30T00:18:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.928+0000 7f5395f5b700 1 -- start start 2023-09-30T00:18:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.928+0000 7f5395f5b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5390070c50 0x7f53900a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f5395f5b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53900b7160 0x7f53900a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f5395f5b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f539013fd70 0x7f53901401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f5395f5b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53900a1310 con 0x7f539013fd70 2023-09-30T00:18:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f5395f5b700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5390071020 con 0x7f5390070c50 2023-09-30T00:18:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f5395f5b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53900b76a0 con 0x7f53900b7160 2023-09-30T00:18:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538ffff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f539013fd70 0x7f53901401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538effd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53900b7160 0x7f53900a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538ffff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f539013fd70 0x7f53901401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36292/0 (socket says 172.21.15.94:36292) 2023-09-30T00:18:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538effd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53900b7160 0x7f53900a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:37384/0 (socket says 172.21.15.94:37384) 2023-09-30T00:18:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538ffff700 1 -- 172.21.15.94:0/2446437199 learned_addr learned my addr 172.21.15.94:0/2446437199 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538f7fe700 1 --2- 172.21.15.94:0/2446437199 >> v2:172.21.15.121:3300/0 conn(0x7f5390070c50 0x7f53900a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538ffff700 1 -- 172.21.15.94:0/2446437199 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53900b7160 msgr2=0x7f53900a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538ffff700 1 --2- 172.21.15.94:0/2446437199 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53900b7160 0x7f53900a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538ffff700 1 -- 172.21.15.94:0/2446437199 >> v2:172.21.15.121:3300/0 conn(0x7f5390070c50 msgr2=0x7f53900a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:12.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538ffff700 1 --2- 172.21.15.94:0/2446437199 >> v2:172.21.15.121:3300/0 conn(0x7f5390070c50 0x7f53900a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:12.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538ffff700 1 -- 172.21.15.94:0/2446437199 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53900b7940 con 0x7f539013fd70 2023-09-30T00:18:12.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538effd700 1 --2- 172.21.15.94:0/2446437199 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53900b7160 0x7f53900a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:12.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538ffff700 1 --2- 172.21.15.94:0/2446437199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f539013fd70 0x7f53901401d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f53900a9760 tx=0x7f5388009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9b7a521e8af9f7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:12.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538e7fc700 1 -- 172.21.15.94:0/2446437199 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388009e70 con 0x7f539013fd70 2023-09-30T00:18:12.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538e7fc700 1 -- 172.21.15.94:0/2446437199 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538800b8f0 con 0x7f539013fd70 2023-09-30T00:18:12.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.929+0000 7f538e7fc700 1 -- 172.21.15.94:0/2446437199 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538800bc90 con 0x7f539013fd70 2023-09-30T00:18:12.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.930+0000 7f5395f5b700 1 -- 172.21.15.94:0/2446437199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f539013fd70 msgr2=0x7f53901401d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:12.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.930+0000 7f5395f5b700 1 --2- 172.21.15.94:0/2446437199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f539013fd70 0x7f53901401d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f53900a9760 tx=0x7f5388009250 comp rx=0 tx=0).stop 2023-09-30T00:18:12.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.930+0000 7f5395f5b700 1 -- 172.21.15.94:0/2446437199 shutdown_connections 2023-09-30T00:18:12.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.930+0000 7f5395f5b700 1 --2- 172.21.15.94:0/2446437199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f539013fd70 0x7f53901401d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:12.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.930+0000 7f5395f5b700 1 --2- 172.21.15.94:0/2446437199 >> v2:172.21.15.121:3300/0 conn(0x7f5390070c50 0x7f53900a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:12.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.930+0000 7f5395f5b700 1 --2- 172.21.15.94:0/2446437199 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53900b7160 0x7f53900a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:12.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.930+0000 7f5395f5b700 1 -- 172.21.15.94:0/2446437199 >> 172.21.15.94:0/2446437199 conn(0x7f539006a090 msgr2=0x7f53900702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:12.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.930+0000 7f5395f5b700 1 -- 172.21.15.94:0/2446437199 shutdown_connections 2023-09-30T00:18:12.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.930+0000 7f5395f5b700 1 -- 172.21.15.94:0/2446437199 wait complete. 2023-09-30T00:18:12.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.931+0000 7f5395f5b700 1 Processor -- start 2023-09-30T00:18:12.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.931+0000 7f5395f5b700 1 -- start start 2023-09-30T00:18:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f5395f5b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5390070c50 0x7f53900abc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:12.939 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f5395f5b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53900b7160 0x7f5390154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:12.939 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f5395f5b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f539013fd70 0x7f5390154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:12.939 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f5395f5b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900ab1c0 con 0x7f53900b7160 2023-09-30T00:18:12.939 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f5395f5b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900ab4a0 con 0x7f539013fd70 2023-09-30T00:18:12.939 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f5395f5b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53900ac390 con 0x7f5390070c50 2023-09-30T00:18:12.939 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f538f7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5390070c50 0x7f53900abc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:12.939 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f538effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53900b7160 0x7f5390154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:12.939 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f538f7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5390070c50 0x7f53900abc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:37396/0 (socket says 172.21.15.94:37396) 2023-09-30T00:18:12.939 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f538f7fe700 1 -- 172.21.15.94:0/1435917256 learned_addr learned my addr 172.21.15.94:0/1435917256 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:12.940 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f538ffff700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f539013fd70 0x7f5390154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:12.940 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f538effd700 1 -- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5390070c50 msgr2=0x7f53900abc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:12.940 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f538effd700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5390070c50 0x7f53900abc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:12.940 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f538effd700 1 -- 172.21.15.94:0/1435917256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f539013fd70 msgr2=0x7f5390154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:12.940 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f538effd700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f539013fd70 0x7f5390154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:12.940 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f538effd700 1 -- 172.21.15.94:0/1435917256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5384009420 con 0x7f53900b7160 2023-09-30T00:18:12.940 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f538ffff700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f539013fd70 0x7f5390154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:18:12.940 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f538effd700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53900b7160 0x7f5390154040 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f538400ec80 tx=0x7f53840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:12.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f53777fe700 1 -- 172.21.15.94:0/1435917256 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538400c850 con 0x7f53900b7160 2023-09-30T00:18:12.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f53777fe700 1 -- 172.21.15.94:0/1435917256 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538400c9b0 con 0x7f53900b7160 2023-09-30T00:18:12.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f5395f5b700 1 -- 172.21.15.94:0/1435917256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53880094f0 con 0x7f53900b7160 2023-09-30T00:18:12.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f53777fe700 1 -- 172.21.15.94:0/1435917256 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538400cd50 con 0x7f53900b7160 2023-09-30T00:18:12.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.932+0000 7f5395f5b700 1 -- 172.21.15.94:0/1435917256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539013d7d0 con 0x7f53900b7160 2023-09-30T00:18:12.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.933+0000 7f5395f5b700 1 -- 172.21.15.94:0/1435917256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53900b7b50 con 0x7f53900b7160 2023-09-30T00:18:12.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.934+0000 7f53777fe700 1 -- 172.21.15.94:0/1435917256 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5384013680 con 0x7f53900b7160 2023-09-30T00:18:12.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.934+0000 7f53777fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:12.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.934+0000 7f53777fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:12.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.934+0000 7f53777fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:12.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.934+0000 7f53777fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:12.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.934+0000 7f53777fe700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f537c0987b0 0x7f537c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:12.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.935+0000 7f538f7fe700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f537c0987b0 0x7f537c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:12.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.935+0000 7f53777fe700 1 -- 172.21.15.94:0/1435917256 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f53840c06e0 con 0x7f53900b7160 2023-09-30T00:18:12.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.935+0000 7f538f7fe700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f537c0987b0 0x7f537c09ac60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5380007700 tx=0x7f5380009920 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:12.943 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:12.939+0000 7f53777fe700 1 -- 172.21.15.94:0/1435917256 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f538408dd90 con 0x7f53900b7160 2023-09-30T00:18:13.164 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.164+0000 7f5395f5b700 1 -- 172.21.15.94:0/1435917256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f539013e180 con 0x7f53900b7160 2023-09-30T00:18:13.165 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.164+0000 7f53777fe700 1 -- 172.21.15.94:0/1435917256 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5384017100 con 0x7f53900b7160 2023-09-30T00:18:13.165 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:13.165 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:13.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 10 mgrc shutdown 2023-09-30T00:18:13.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 1 -- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f537c0987b0 msgr2=0x7f537c09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:13.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f537c0987b0 0x7f537c09ac60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5380007700 tx=0x7f5380009920 comp rx=0 tx=0).stop 2023-09-30T00:18:13.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 1 -- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53900b7160 msgr2=0x7f5390154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:13.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53900b7160 0x7f5390154040 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f538400ec80 tx=0x7f53840091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:13.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 1 -- 172.21.15.94:0/1435917256 shutdown_connections 2023-09-30T00:18:13.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f537c0987b0 0x7f537c09ac60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:13.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5390070c50 0x7f53900abc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:13.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f539013fd70 0x7f5390154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:13.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 1 --2- 172.21.15.94:0/1435917256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53900b7160 0x7f5390154040 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:13.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 1 -- 172.21.15.94:0/1435917256 >> 172.21.15.94:0/1435917256 conn(0x7f539006a090 msgr2=0x7f539013ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:13.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 1 -- 172.21.15.94:0/1435917256 shutdown_connections 2023-09-30T00:18:13.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:13.166+0000 7f5395f5b700 1 -- 172.21.15.94:0/1435917256 wait complete. 2023-09-30T00:18:16.180 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.303+0000 7ff94bfff700 1 Processor -- start 2023-09-30T00:18:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.303+0000 7ff94bfff700 1 -- start start 2023-09-30T00:18:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.304+0000 7ff94bfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 0x7ff94c1419d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.304+0000 7ff94bfff700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c0a73d0 con 0x7ff94c141f10 2023-09-30T00:18:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.304+0000 7ff94bfff700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c142920 con 0x7ff94c0a7000 2023-09-30T00:18:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.304+0000 7ff94bfff700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c142660 con 0x7ff94c0b4e80 2023-09-30T00:18:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.304+0000 7ff9427fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 0x7ff94c1419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.304+0000 7ff94b7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff94c141f10 0x7ff94c148d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50178/0 (socket says 172.21.15.94:50178) 2023-09-30T00:18:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.304+0000 7ff94b7fe700 1 -- 172.21.15.94:0/1135780562 learned_addr learned my addr 172.21.15.94:0/1135780562 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.304+0000 7ff9427fc700 1 -- 172.21.15.94:0/1135780562 >> v1:172.21.15.121:6789/0 conn(0x7ff94c0a7000 legacy=0x7ff94c134270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.304+0000 7ff9427fc700 1 -- 172.21.15.94:0/1135780562 >> v1:172.21.15.94:6789/0 conn(0x7ff94c141f10 legacy=0x7ff94c148d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.304+0000 7ff94a7fc700 1 -- 172.21.15.94:0/1135780562 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077440105 0 0) 0x7ff94c0a73d0 con 0x7ff94c141f10 2023-09-30T00:18:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.304+0000 7ff9427fc700 1 -- 172.21.15.94:0/1135780562 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c142390 con 0x7ff94c0b4e80 2023-09-30T00:18:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.305+0000 7ff9427fc700 1 --2- 172.21.15.94:0/1135780562 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 0x7ff94c1419d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff934009910 tx=0x7ff9340091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77d17a6bd4346a5e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.305+0000 7ff94a7fc700 1 -- 172.21.15.94:0/1135780562 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93400b860 con 0x7ff94c0b4e80 2023-09-30T00:18:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.305+0000 7ff94a7fc700 1 -- 172.21.15.94:0/1135780562 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93400b9c0 con 0x7ff94c0b4e80 2023-09-30T00:18:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.305+0000 7ff94a7fc700 1 -- 172.21.15.94:0/1135780562 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93400bd60 con 0x7ff94c0b4e80 2023-09-30T00:18:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.305+0000 7ff94bfff700 1 -- 172.21.15.94:0/1135780562 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 msgr2=0x7ff94c1419d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.305+0000 7ff94bfff700 1 --2- 172.21.15.94:0/1135780562 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 0x7ff94c1419d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff934009910 tx=0x7ff9340091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:16.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.305+0000 7ff94bfff700 1 -- 172.21.15.94:0/1135780562 shutdown_connections 2023-09-30T00:18:16.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.305+0000 7ff94bfff700 1 --2- 172.21.15.94:0/1135780562 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 0x7ff94c1419d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:16.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.305+0000 7ff94bfff700 1 -- 172.21.15.94:0/1135780562 >> 172.21.15.94:0/1135780562 conn(0x7ff94c06c940 msgr2=0x7ff94c06cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:16.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.305+0000 7ff94bfff700 1 -- 172.21.15.94:0/1135780562 shutdown_connections 2023-09-30T00:18:16.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.305+0000 7ff94bfff700 1 -- 172.21.15.94:0/1135780562 wait complete. 2023-09-30T00:18:16.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.306+0000 7ff94bfff700 1 Processor -- start 2023-09-30T00:18:16.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.306+0000 7ff94bfff700 1 -- start start 2023-09-30T00:18:16.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff94bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff94c0a7000 0x7ff94c0ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:16.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff94bfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 0x7ff94c0af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:16.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff94bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff94c141f10 0x7ff94c12fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:16.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff94bfff700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c12f970 con 0x7ff94c0a7000 2023-09-30T00:18:16.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff94bfff700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0ae6d0 con 0x7ff94c141f10 2023-09-30T00:18:16.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff94bfff700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0afd40 con 0x7ff94c0b4e80 2023-09-30T00:18:16.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff9427fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 0x7ff94c0af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:16.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff94affd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff94c0a7000 0x7ff94c0ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:16.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff9427fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 0x7ff94c0af800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:40306/0 (socket says 172.21.15.94:40306) 2023-09-30T00:18:16.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff9427fc700 1 -- 172.21.15.94:0/1535105713 learned_addr learned my addr 172.21.15.94:0/1535105713 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:16.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff94b7fe700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff94c141f10 0x7ff94c12fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:16.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff9427fc700 1 -- 172.21.15.94:0/1535105713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff94c141f10 msgr2=0x7ff94c12fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:16.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff9427fc700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff94c141f10 0x7ff94c12fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:16.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff9427fc700 1 -- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff94c0a7000 msgr2=0x7ff94c0ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:16.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff9427fc700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff94c0a7000 0x7ff94c0ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:16.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff9427fc700 1 -- 172.21.15.94:0/1535105713 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff934009420 con 0x7ff94c0b4e80 2023-09-30T00:18:16.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.307+0000 7ff94affd700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff94c0a7000 0x7ff94c0ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.308+0000 7ff9427fc700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 0x7ff94c0af800 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff94c0b5920 tx=0x7ff934017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.308+0000 7ff9437fe700 1 -- 172.21.15.94:0/1535105713 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93401f070 con 0x7ff94c0b4e80 2023-09-30T00:18:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.308+0000 7ff9437fe700 1 -- 172.21.15.94:0/1535105713 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff934002f70 con 0x7ff94c0b4e80 2023-09-30T00:18:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.308+0000 7ff94bfff700 1 -- 172.21.15.94:0/1535105713 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94c0b0170 con 0x7ff94c0b4e80 2023-09-30T00:18:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.308+0000 7ff9437fe700 1 -- 172.21.15.94:0/1535105713 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93401a670 con 0x7ff94c0b4e80 2023-09-30T00:18:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.308+0000 7ff94bfff700 1 -- 172.21.15.94:0/1535105713 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94c130320 con 0x7ff94c0b4e80 2023-09-30T00:18:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.309+0000 7ff94bfff700 1 -- 172.21.15.94:0/1535105713 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94c04c380 con 0x7ff94c0b4e80 2023-09-30T00:18:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.309+0000 7ff9437fe700 1 -- 172.21.15.94:0/1535105713 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ff93401aa20 con 0x7ff94c0b4e80 2023-09-30T00:18:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.309+0000 7ff9437fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.310+0000 7ff9437fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.310+0000 7ff9437fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:16.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.310+0000 7ff9437fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:16.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.310+0000 7ff9437fe700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff92c0987b0 0x7ff92c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:16.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.310+0000 7ff94affd700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff92c0987b0 0x7ff92c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:16.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.310+0000 7ff9437fe700 1 -- 172.21.15.94:0/1535105713 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7ff9340c1410 con 0x7ff94c0b4e80 2023-09-30T00:18:16.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.310+0000 7ff94affd700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff92c0987b0 0x7ff92c09ac60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff93c001dc0 tx=0x7ff93c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:16.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.314+0000 7ff9437fe700 1 -- 172.21.15.94:0/1535105713 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff93408eb50 con 0x7ff94c0b4e80 2023-09-30T00:18:16.538 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.537+0000 7ff94bfff700 1 -- 172.21.15.94:0/1535105713 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff94c131c60 con 0x7ff94c0b4e80 2023-09-30T00:18:16.538 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.538+0000 7ff9437fe700 1 -- 172.21.15.94:0/1535105713 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff934015100 con 0x7ff94c0b4e80 2023-09-30T00:18:16.538 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:16.538 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:16.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.539+0000 7ff94bfff700 10 mgrc shutdown 2023-09-30T00:18:16.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.539+0000 7ff94bfff700 1 -- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff92c0987b0 msgr2=0x7ff92c09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:16.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.539+0000 7ff94bfff700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff92c0987b0 0x7ff92c09ac60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff93c001dc0 tx=0x7ff93c009040 comp rx=0 tx=0).stop 2023-09-30T00:18:16.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.539+0000 7ff94bfff700 1 -- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 msgr2=0x7ff94c0af800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:16.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.539+0000 7ff94bfff700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 0x7ff94c0af800 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff94c0b5920 tx=0x7ff934017000 comp rx=0 tx=0).stop 2023-09-30T00:18:16.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.540+0000 7ff94bfff700 1 -- 172.21.15.94:0/1535105713 shutdown_connections 2023-09-30T00:18:16.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.540+0000 7ff94bfff700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff92c0987b0 0x7ff92c09ac60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:16.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.540+0000 7ff94bfff700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff94c141f10 0x7ff94c12fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:16.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.540+0000 7ff94bfff700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff94c0a7000 0x7ff94c0ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:16.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.540+0000 7ff94bfff700 1 --2- 172.21.15.94:0/1535105713 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff94c0b4e80 0x7ff94c0af800 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:16.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.540+0000 7ff94bfff700 1 -- 172.21.15.94:0/1535105713 >> 172.21.15.94:0/1535105713 conn(0x7ff94c06c940 msgr2=0x7ff94c141690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:16.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.540+0000 7ff94bfff700 1 -- 172.21.15.94:0/1535105713 shutdown_connections 2023-09-30T00:18:16.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:16.540+0000 7ff94bfff700 1 -- 172.21.15.94:0/1535105713 wait complete. 2023-09-30T00:18:19.552 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.677+0000 7fc6d2595700 1 Processor -- start 2023-09-30T00:18:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.677+0000 7fc6d2595700 1 -- start start 2023-09-30T00:18:19.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.677+0000 7fc6d2595700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc6cc0a46c0 0x7fc6cc068c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:19.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.677+0000 7fc6d2595700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc0b5850 0x7fc6cc14b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:19.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.677+0000 7fc6d2595700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6cc069760 con 0x7fc6cc06a460 2023-09-30T00:18:19.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.677+0000 7fc6d2595700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6cc0a57f0 con 0x7fc6cc0a46c0 2023-09-30T00:18:19.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.677+0000 7fc6d2595700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc0a4a90 con 0x7fc6cc0b5850 2023-09-30T00:18:19.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.677+0000 7fc6d0b32700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc0b5850 0x7fc6cc14b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:19.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.677+0000 7fc6cb7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc6cc06a460 0x7fc6cc0b5140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50188/0 (socket says 172.21.15.94:50188) 2023-09-30T00:18:19.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.677+0000 7fc6cb7fe700 1 -- 172.21.15.94:0/2567684779 learned_addr learned my addr 172.21.15.94:0/2567684779 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:19.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6cbfff700 1 --2- 172.21.15.94:0/2567684779 >> v2:172.21.15.121:3300/0 conn(0x7fc6cc0a46c0 0x7fc6cc068c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:19.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6caffd700 1 -- 172.21.15.94:0/2567684779 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154333178 0 0) 0x7fc6cc069760 con 0x7fc6cc06a460 2023-09-30T00:18:19.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6d0b32700 1 -- 172.21.15.94:0/2567684779 >> v2:172.21.15.121:3300/0 conn(0x7fc6cc0a46c0 msgr2=0x7fc6cc068c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6d0b32700 1 --2- 172.21.15.94:0/2567684779 >> v2:172.21.15.121:3300/0 conn(0x7fc6cc0a46c0 0x7fc6cc068c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6d0b32700 1 -- 172.21.15.94:0/2567684779 >> v1:172.21.15.94:6789/0 conn(0x7fc6cc06a460 legacy=0x7fc6cc0b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6d0b32700 1 -- 172.21.15.94:0/2567684779 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc069c80 con 0x7fc6cc0b5850 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6cbfff700 1 --2- 172.21.15.94:0/2567684779 >> v2:172.21.15.121:3300/0 conn(0x7fc6cc0a46c0 0x7fc6cc068c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6d0b32700 1 --2- 172.21.15.94:0/2567684779 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc0b5850 0x7fc6cc14b060 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009e90 tx=0x7fc6c4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e01966c046efab81 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6caffd700 1 -- 172.21.15.94:0/2567684779 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c400e070 con 0x7fc6cc0b5850 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6caffd700 1 -- 172.21.15.94:0/2567684779 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c400bbf0 con 0x7fc6cc0b5850 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6caffd700 1 -- 172.21.15.94:0/2567684779 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4012430 con 0x7fc6cc0b5850 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6d2595700 1 -- 172.21.15.94:0/2567684779 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc0b5850 msgr2=0x7fc6cc14b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6d2595700 1 --2- 172.21.15.94:0/2567684779 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc0b5850 0x7fc6cc14b060 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009e90 tx=0x7fc6c4009780 comp rx=0 tx=0).stop 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6d2595700 1 -- 172.21.15.94:0/2567684779 shutdown_connections 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6d2595700 1 --2- 172.21.15.94:0/2567684779 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc0b5850 0x7fc6cc14b060 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:19.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6d2595700 1 --2- 172.21.15.94:0/2567684779 >> v2:172.21.15.121:3300/0 conn(0x7fc6cc0a46c0 0x7fc6cc068c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.678+0000 7fc6d2595700 1 -- 172.21.15.94:0/2567684779 >> 172.21.15.94:0/2567684779 conn(0x7fc6cc06cbf0 msgr2=0x7fc6cc06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.679+0000 7fc6d2595700 1 -- 172.21.15.94:0/2567684779 shutdown_connections 2023-09-30T00:18:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.679+0000 7fc6d2595700 1 -- 172.21.15.94:0/2567684779 wait complete. 2023-09-30T00:18:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.679+0000 7fc6d2595700 1 Processor -- start 2023-09-30T00:18:19.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.680+0000 7fc6d2595700 1 -- start start 2023-09-30T00:18:19.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.680+0000 7fc6d2595700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc06a460 0x7fc6cc1459c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:19.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.680+0000 7fc6d2595700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6cc0a46c0 0x7fc6cc13ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:19.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.680+0000 7fc6d2595700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6cc0b5850 0x7fc6cc1404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:19.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.680+0000 7fc6d2595700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc1409f0 con 0x7fc6cc0a46c0 2023-09-30T00:18:19.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.680+0000 7fc6d2595700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc1441e0 con 0x7fc6cc0b5850 2023-09-30T00:18:19.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.680+0000 7fc6d2595700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc144410 con 0x7fc6cc06a460 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.680+0000 7fc6cbfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc06a460 0x7fc6cc1459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.680+0000 7fc6cbfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc06a460 0x7fc6cc1459c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:40324/0 (socket says 172.21.15.94:40324) 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.680+0000 7fc6cbfff700 1 -- 172.21.15.94:0/998068203 learned_addr learned my addr 172.21.15.94:0/998068203 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.680+0000 7fc6cb7fe700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6cc0a46c0 0x7fc6cc13ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.681+0000 7fc6d0b32700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6cc0b5850 0x7fc6cc1404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.681+0000 7fc6cbfff700 1 -- 172.21.15.94:0/998068203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6cc0b5850 msgr2=0x7fc6cc1404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.681+0000 7fc6cbfff700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6cc0b5850 0x7fc6cc1404b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.681+0000 7fc6cbfff700 1 -- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6cc0a46c0 msgr2=0x7fc6cc13ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.681+0000 7fc6cbfff700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6cc0a46c0 0x7fc6cc13ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.681+0000 7fc6cbfff700 1 -- 172.21.15.94:0/998068203 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c4009a20 con 0x7fc6cc06a460 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.681+0000 7fc6cbfff700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc06a460 0x7fc6cc1459c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc00e8d0 tx=0x7fc6bc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.681+0000 7fc6b3fff700 1 -- 172.21.15.94:0/998068203 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc013410 con 0x7fc6cc06a460 2023-09-30T00:18:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.681+0000 7fc6b3fff700 1 -- 172.21.15.94:0/998068203 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6bc013570 con 0x7fc6cc06a460 2023-09-30T00:18:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.681+0000 7fc6d2595700 1 -- 172.21.15.94:0/998068203 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6cc13f9b0 con 0x7fc6cc06a460 2023-09-30T00:18:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.681+0000 7fc6b3fff700 1 -- 172.21.15.94:0/998068203 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc013910 con 0x7fc6cc06a460 2023-09-30T00:18:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.681+0000 7fc6d2595700 1 -- 172.21.15.94:0/998068203 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6cc1436d0 con 0x7fc6cc06a460 2023-09-30T00:18:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.682+0000 7fc6d2595700 1 -- 172.21.15.94:0/998068203 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6cc06e6f0 con 0x7fc6cc06a460 2023-09-30T00:18:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.683+0000 7fc6b3fff700 1 -- 172.21.15.94:0/998068203 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc6bc013d00 con 0x7fc6cc06a460 2023-09-30T00:18:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.683+0000 7fc6b3fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.683+0000 7fc6b3fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.683+0000 7fc6b3fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.683+0000 7fc6b3fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.683+0000 7fc6b3fff700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc6b8098780 0x7fc6b809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.683+0000 7fc6cb7fe700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc6b8098780 0x7fc6b809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.683+0000 7fc6b3fff700 1 -- 172.21.15.94:0/998068203 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fc6bc0c0580 con 0x7fc6cc06a460 2023-09-30T00:18:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.684+0000 7fc6cb7fe700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc6b8098780 0x7fc6b809ac30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc146500 tx=0x7fc6c0007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.687+0000 7fc6b3fff700 1 -- 172.21.15.94:0/998068203 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc6bc091750 con 0x7fc6cc06a460 2023-09-30T00:18:19.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.912+0000 7fc6d2595700 1 -- 172.21.15.94:0/998068203 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc6cc13fba0 con 0x7fc6cc06a460 2023-09-30T00:18:19.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.912+0000 7fc6b3fff700 1 -- 172.21.15.94:0/998068203 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc6bc020100 con 0x7fc6cc06a460 2023-09-30T00:18:19.913 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:19.913 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:19.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.914+0000 7fc6d2595700 10 mgrc shutdown 2023-09-30T00:18:19.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.915+0000 7fc6d2595700 1 -- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc6b8098780 msgr2=0x7fc6b809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:19.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.915+0000 7fc6d2595700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc6b8098780 0x7fc6b809ac30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc146500 tx=0x7fc6c0007d30 comp rx=0 tx=0).stop 2023-09-30T00:18:19.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.915+0000 7fc6d2595700 1 -- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc06a460 msgr2=0x7fc6cc1459c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:19.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.915+0000 7fc6d2595700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc06a460 0x7fc6cc1459c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc00e8d0 tx=0x7fc6bc008040 comp rx=0 tx=0).stop 2023-09-30T00:18:19.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.915+0000 7fc6d2595700 1 -- 172.21.15.94:0/998068203 shutdown_connections 2023-09-30T00:18:19.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.915+0000 7fc6d2595700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc6b8098780 0x7fc6b809ac30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:19.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.915+0000 7fc6d2595700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6cc06a460 0x7fc6cc1459c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:19.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.915+0000 7fc6d2595700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6cc0b5850 0x7fc6cc1404b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:19.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.915+0000 7fc6d2595700 1 --2- 172.21.15.94:0/998068203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6cc0a46c0 0x7fc6cc13ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:19.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.915+0000 7fc6d2595700 1 -- 172.21.15.94:0/998068203 >> 172.21.15.94:0/998068203 conn(0x7fc6cc06cbf0 msgr2=0x7fc6cc0b2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:19.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.915+0000 7fc6d2595700 1 -- 172.21.15.94:0/998068203 shutdown_connections 2023-09-30T00:18:19.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:19.915+0000 7fc6d2595700 1 -- 172.21.15.94:0/998068203 wait complete. 2023-09-30T00:18:22.928 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:23.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.051+0000 7fb6168e9700 1 Processor -- start 2023-09-30T00:18:23.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.051+0000 7fb6168e9700 1 -- start start 2023-09-30T00:18:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.052+0000 7fb6168e9700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb6100a4470 0x7fb6100b5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.052+0000 7fb6168e9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4840 0x7fb6100a5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.052+0000 7fb6168e9700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6100a6050 con 0x7fb6100a6fc0 2023-09-30T00:18:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.052+0000 7fb6168e9700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6100b4dd0 con 0x7fb6100a4470 2023-09-30T00:18:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.052+0000 7fb6168e9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb61006ea50 con 0x7fb6100a4840 2023-09-30T00:18:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.052+0000 7fb60f7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4840 0x7fb6100a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.052+0000 7fb614e86700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb6100a6fc0 0x7fb610141a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50192/0 (socket says 172.21.15.94:50192) 2023-09-30T00:18:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.052+0000 7fb614e86700 1 -- 172.21.15.94:0/604736116 learned_addr learned my addr 172.21.15.94:0/604736116 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.052+0000 7fb60ffff700 1 --2- 172.21.15.94:0/604736116 >> v2:172.21.15.121:3300/0 conn(0x7fb6100a4470 0x7fb6100b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.052+0000 7fb60effd700 1 -- 172.21.15.94:0/604736116 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921606254 0 0) 0x7fb6100a6050 con 0x7fb6100a6fc0 2023-09-30T00:18:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.052+0000 7fb60effd700 1 -- 172.21.15.94:0/604736116 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5f8003570 con 0x7fb6100a6fc0 2023-09-30T00:18:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb60f7fe700 1 -- 172.21.15.94:0/604736116 >> v2:172.21.15.121:3300/0 conn(0x7fb6100a4470 msgr2=0x7fb6100b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb60f7fe700 1 --2- 172.21.15.94:0/604736116 >> v2:172.21.15.121:3300/0 conn(0x7fb6100a4470 0x7fb6100b5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb60f7fe700 1 -- 172.21.15.94:0/604736116 >> v1:172.21.15.94:6789/0 conn(0x7fb6100a6fc0 legacy=0x7fb610141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:23.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb60f7fe700 1 -- 172.21.15.94:0/604736116 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6100b5a90 con 0x7fb6100a4840 2023-09-30T00:18:23.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb60ffff700 1 --2- 172.21.15.94:0/604736116 >> v2:172.21.15.121:3300/0 conn(0x7fb6100a4470 0x7fb6100b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:23.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb60f7fe700 1 --2- 172.21.15.94:0/604736116 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4840 0x7fb6100a5720 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb6040098e0 tx=0x7fb6040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf7d450d748eb003 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:23.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb60effd700 1 -- 172.21.15.94:0/604736116 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60400b640 con 0x7fb6100a4840 2023-09-30T00:18:23.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb60effd700 1 -- 172.21.15.94:0/604736116 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb60400b7a0 con 0x7fb6100a4840 2023-09-30T00:18:23.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb60effd700 1 -- 172.21.15.94:0/604736116 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60400bb40 con 0x7fb6100a4840 2023-09-30T00:18:23.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb6168e9700 1 -- 172.21.15.94:0/604736116 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4840 msgr2=0x7fb6100a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:23.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb6168e9700 1 --2- 172.21.15.94:0/604736116 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4840 0x7fb6100a5720 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb6040098e0 tx=0x7fb6040091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:23.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb6168e9700 1 -- 172.21.15.94:0/604736116 shutdown_connections 2023-09-30T00:18:23.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb6168e9700 1 --2- 172.21.15.94:0/604736116 >> v2:172.21.15.121:3300/0 conn(0x7fb6100a4470 0x7fb6100b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:23.057 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb6168e9700 1 --2- 172.21.15.94:0/604736116 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4840 0x7fb6100a5720 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:23.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.053+0000 7fb6168e9700 1 -- 172.21.15.94:0/604736116 >> 172.21.15.94:0/604736116 conn(0x7fb61006cbf0 msgr2=0x7fb61006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:23.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.054+0000 7fb6168e9700 1 -- 172.21.15.94:0/604736116 shutdown_connections 2023-09-30T00:18:23.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.054+0000 7fb6168e9700 1 -- 172.21.15.94:0/604736116 wait complete. 2023-09-30T00:18:23.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.054+0000 7fb6168e9700 1 Processor -- start 2023-09-30T00:18:23.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.055+0000 7fb6168e9700 1 -- start start 2023-09-30T00:18:23.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.055+0000 7fb6168e9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4470 0x7fb610141d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:23.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.055+0000 7fb6168e9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6100a4840 0x7fb6100af030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:23.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.055+0000 7fb6168e9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6100a6fc0 0x7fb610153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:23.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.055+0000 7fb6168e9700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6100afa40 con 0x7fb6100a6fc0 2023-09-30T00:18:23.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.055+0000 7fb6168e9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6100ae220 con 0x7fb6100a4840 2023-09-30T00:18:23.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.055+0000 7fb6168e9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb610069e00 con 0x7fb6100a4470 2023-09-30T00:18:23.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.055+0000 7fb60ffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4470 0x7fb610141d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:23.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.055+0000 7fb60ffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4470 0x7fb610141d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:40338/0 (socket says 172.21.15.94:40338) 2023-09-30T00:18:23.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.055+0000 7fb60ffff700 1 -- 172.21.15.94:0/3186684773 learned_addr learned my addr 172.21.15.94:0/3186684773 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:23.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.055+0000 7fb614e86700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6100a6fc0 0x7fb610153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:23.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.055+0000 7fb60f7fe700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6100a4840 0x7fb6100af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:23.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.056+0000 7fb60ffff700 1 -- 172.21.15.94:0/3186684773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6100a4840 msgr2=0x7fb6100af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:23.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.056+0000 7fb60ffff700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6100a4840 0x7fb6100af030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:23.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.056+0000 7fb60ffff700 1 -- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6100a6fc0 msgr2=0x7fb610153630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:23.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.056+0000 7fb60ffff700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6100a6fc0 0x7fb610153630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:23.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.056+0000 7fb60ffff700 1 -- 172.21.15.94:0/3186684773 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb604009420 con 0x7fb6100a4470 2023-09-30T00:18:23.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.056+0000 7fb614e86700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6100a6fc0 0x7fb610153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:23.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.056+0000 7fb60ffff700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4470 0x7fb610141d30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb60000d860 tx=0x7fb600008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:23.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.056+0000 7fb5f7fff700 1 -- 172.21.15.94:0/3186684773 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6000125f0 con 0x7fb6100a4470 2023-09-30T00:18:23.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.056+0000 7fb5f7fff700 1 -- 172.21.15.94:0/3186684773 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb600012750 con 0x7fb6100a4470 2023-09-30T00:18:23.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.056+0000 7fb6168e9700 1 -- 172.21.15.94:0/3186684773 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb610142490 con 0x7fb6100a4470 2023-09-30T00:18:23.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.056+0000 7fb5f7fff700 1 -- 172.21.15.94:0/3186684773 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb600012af0 con 0x7fb6100a4470 2023-09-30T00:18:23.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.056+0000 7fb6168e9700 1 -- 172.21.15.94:0/3186684773 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6100af730 con 0x7fb6100a4470 2023-09-30T00:18:23.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.057+0000 7fb6168e9700 1 -- 172.21.15.94:0/3186684773 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb61006fbd0 con 0x7fb6100a4470 2023-09-30T00:18:23.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.058+0000 7fb5f7fff700 1 -- 172.21.15.94:0/3186684773 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb600014430 con 0x7fb6100a4470 2023-09-30T00:18:23.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.058+0000 7fb5f7fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:23.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.058+0000 7fb5f7fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:23.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.058+0000 7fb5f7fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:23.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.058+0000 7fb5f7fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:23.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.058+0000 7fb5f7fff700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb5fc098780 0x7fb5fc09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:23.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.058+0000 7fb60f7fe700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb5fc098780 0x7fb5fc09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:23.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.059+0000 7fb5f7fff700 1 -- 172.21.15.94:0/3186684773 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7fb6000c0940 con 0x7fb6100a4470 2023-09-30T00:18:23.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.059+0000 7fb60f7fe700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb5fc098780 0x7fb5fc09ac30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb6040093f0 tx=0x7fb604017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:23.064 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.062+0000 7fb5f7fff700 1 -- 172.21.15.94:0/3186684773 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb60008df50 con 0x7fb6100a4470 2023-09-30T00:18:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.287+0000 7fb6168e9700 1 -- 172.21.15.94:0/3186684773 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb610069f40 con 0x7fb6100a4470 2023-09-30T00:18:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.288+0000 7fb5f7fff700 1 -- 172.21.15.94:0/3186684773 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb610069f40 con 0x7fb6100a4470 2023-09-30T00:18:23.289 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:23.289 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 10 mgrc shutdown 2023-09-30T00:18:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 1 -- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb5fc098780 msgr2=0x7fb5fc09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb5fc098780 0x7fb5fc09ac30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb6040093f0 tx=0x7fb604017000 comp rx=0 tx=0).stop 2023-09-30T00:18:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 1 -- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4470 msgr2=0x7fb610141d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4470 0x7fb610141d30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb60000d860 tx=0x7fb600008040 comp rx=0 tx=0).stop 2023-09-30T00:18:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 1 -- 172.21.15.94:0/3186684773 shutdown_connections 2023-09-30T00:18:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb5fc098780 0x7fb5fc09ac30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6100a4470 0x7fb610141d30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6100a6fc0 0x7fb610153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 1 --2- 172.21.15.94:0/3186684773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6100a4840 0x7fb6100af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:23.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 1 -- 172.21.15.94:0/3186684773 >> 172.21.15.94:0/3186684773 conn(0x7fb61006cbf0 msgr2=0x7fb61006cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 1 -- 172.21.15.94:0/3186684773 shutdown_connections 2023-09-30T00:18:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:23.290+0000 7fb6168e9700 1 -- 172.21.15.94:0/3186684773 wait complete. 2023-09-30T00:18:26.302 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:26.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.427+0000 7f943e145700 1 Processor -- start 2023-09-30T00:18:26.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.427+0000 7f943e145700 1 -- start start 2023-09-30T00:18:26.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.427+0000 7f943e145700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 0x7f94380a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:26.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.427+0000 7f943e145700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f94380b7160 0x7f94380a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:26.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.427+0000 7f943e145700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94380a1310 con 0x7f943813fd70 2023-09-30T00:18:26.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.427+0000 7f943e145700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9438071020 con 0x7f94380b7160 2023-09-30T00:18:26.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.427+0000 7f943e145700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94380b76a0 con 0x7f9438070c50 2023-09-30T00:18:26.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f94377fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 0x7f94380a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:26.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f9437fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f943813fd70 0x7f94380b4dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57812/0 (socket says 172.21.15.94:57812) 2023-09-30T00:18:26.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f9437fff700 1 -- 172.21.15.94:0/3790921827 learned_addr learned my addr 172.21.15.94:0/3790921827 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:26.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f9436ffd700 1 --2- 172.21.15.94:0/3790921827 >> v2:172.21.15.121:3300/0 conn(0x7f94380b7160 0x7f94380a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:26.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f94367fc700 1 -- 172.21.15.94:0/3790921827 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182223522 0 0) 0x7f9430001490 con 0x7f943813fd70 2023-09-30T00:18:26.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f94377fe700 1 -- 172.21.15.94:0/3790921827 >> v2:172.21.15.121:3300/0 conn(0x7f94380b7160 msgr2=0x7f94380a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:26.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f94377fe700 1 --2- 172.21.15.94:0/3790921827 >> v2:172.21.15.121:3300/0 conn(0x7f94380b7160 0x7f94380a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:26.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f94377fe700 1 -- 172.21.15.94:0/3790921827 >> v1:172.21.15.94:6789/0 conn(0x7f943813fd70 legacy=0x7f94380b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:26.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f94377fe700 1 -- 172.21.15.94:0/3790921827 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94380b7940 con 0x7f9438070c50 2023-09-30T00:18:26.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f94377fe700 1 --2- 172.21.15.94:0/3790921827 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 0x7f94380a0dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f942c0098e0 tx=0x7f942c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2e0d62dc29a37f9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:26.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f94367fc700 1 -- 172.21.15.94:0/3790921827 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c009e80 con 0x7f9438070c50 2023-09-30T00:18:26.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f94367fc700 1 -- 172.21.15.94:0/3790921827 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942c00b8f0 con 0x7f9438070c50 2023-09-30T00:18:26.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.428+0000 7f94367fc700 1 -- 172.21.15.94:0/3790921827 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c00bc70 con 0x7f9438070c50 2023-09-30T00:18:26.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.429+0000 7f943e145700 1 -- 172.21.15.94:0/3790921827 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 msgr2=0x7f94380a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:26.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.429+0000 7f943e145700 1 --2- 172.21.15.94:0/3790921827 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 0x7f94380a0dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f942c0098e0 tx=0x7f942c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:26.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.429+0000 7f943e145700 1 -- 172.21.15.94:0/3790921827 shutdown_connections 2023-09-30T00:18:26.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.429+0000 7f943e145700 1 --2- 172.21.15.94:0/3790921827 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 0x7f94380a0dd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:26.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.429+0000 7f943e145700 1 --2- 172.21.15.94:0/3790921827 >> v2:172.21.15.121:3300/0 conn(0x7f94380b7160 0x7f94380a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:26.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.429+0000 7f943e145700 1 -- 172.21.15.94:0/3790921827 >> 172.21.15.94:0/3790921827 conn(0x7f943806a090 msgr2=0x7f94380702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:26.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.429+0000 7f943e145700 1 -- 172.21.15.94:0/3790921827 shutdown_connections 2023-09-30T00:18:26.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.429+0000 7f943e145700 1 -- 172.21.15.94:0/3790921827 wait complete. 2023-09-30T00:18:26.439 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.430+0000 7f943e145700 1 Processor -- start 2023-09-30T00:18:26.439 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.430+0000 7f943e145700 1 -- start start 2023-09-30T00:18:26.439 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f943e145700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 0x7f943813bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:26.439 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f943e145700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94380b7160 0x7f94381505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:26.439 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f943e145700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f943813fd70 0x7f9438150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:26.439 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f943e145700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94380a6290 con 0x7f94380b7160 2023-09-30T00:18:26.439 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f943e145700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9438151570 con 0x7f943813fd70 2023-09-30T00:18:26.439 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f943e145700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94380acbd0 con 0x7f9438070c50 2023-09-30T00:18:26.439 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f94377fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 0x7f943813bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:26.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f9436ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94380b7160 0x7f94381505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:26.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f94377fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 0x7f943813bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44834/0 (socket says 172.21.15.94:44834) 2023-09-30T00:18:26.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f94377fe700 1 -- 172.21.15.94:0/2763623842 learned_addr learned my addr 172.21.15.94:0/2763623842 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:26.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f9437fff700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f943813fd70 0x7f9438150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:26.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f94377fe700 1 -- 172.21.15.94:0/2763623842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f943813fd70 msgr2=0x7f9438150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:26.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f94377fe700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f943813fd70 0x7f9438150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:26.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f94377fe700 1 -- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94380b7160 msgr2=0x7f94381505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:26.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f94377fe700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94380b7160 0x7f94381505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:26.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f94377fe700 1 -- 172.21.15.94:0/2763623842 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942c009420 con 0x7f9438070c50 2023-09-30T00:18:26.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f9436ffd700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94380b7160 0x7f94381505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:26.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f94377fe700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 0x7f943813bfe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f942c009f00 tx=0x7f942c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:26.441 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f941f7fe700 1 -- 172.21.15.94:0/2763623842 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c00bc70 con 0x7f9438070c50 2023-09-30T00:18:26.441 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f941f7fe700 1 -- 172.21.15.94:0/2763623842 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942c011420 con 0x7f9438070c50 2023-09-30T00:18:26.441 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f943e145700 1 -- 172.21.15.94:0/2763623842 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94381512b0 con 0x7f9438070c50 2023-09-30T00:18:26.441 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f941f7fe700 1 -- 172.21.15.94:0/2763623842 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c0117c0 con 0x7f9438070c50 2023-09-30T00:18:26.441 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.431+0000 7f943e145700 1 -- 172.21.15.94:0/2763623842 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94380acf60 con 0x7f9438070c50 2023-09-30T00:18:26.441 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.432+0000 7f943e145700 1 -- 172.21.15.94:0/2763623842 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9438071630 con 0x7f9438070c50 2023-09-30T00:18:26.441 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.433+0000 7f941f7fe700 1 -- 172.21.15.94:0/2763623842 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f942c011bb0 con 0x7f9438070c50 2023-09-30T00:18:26.441 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.433+0000 7f941f7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:26.441 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.434+0000 7f941f7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:26.441 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.434+0000 7f941f7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:26.441 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.434+0000 7f941f7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:26.442 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.434+0000 7f941f7fe700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f94240987b0 0x7f942409ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:26.442 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.434+0000 7f9436ffd700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f94240987b0 0x7f942409ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:26.442 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.434+0000 7f941f7fe700 1 -- 172.21.15.94:0/2763623842 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f942c0c2530 con 0x7f9438070c50 2023-09-30T00:18:26.442 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.434+0000 7f9436ffd700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f94240987b0 0x7f942409ac60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f943813d940 tx=0x7f9428008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:26.442 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.437+0000 7f941f7fe700 1 -- 172.21.15.94:0/2763623842 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f942c08fc70 con 0x7f9438070c50 2023-09-30T00:18:26.659 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.658+0000 7f943e145700 1 -- 172.21.15.94:0/2763623842 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f94380a2070 con 0x7f9438070c50 2023-09-30T00:18:26.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.658+0000 7f941f7fe700 1 -- 172.21.15.94:0/2763623842 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f942c015100 con 0x7f9438070c50 2023-09-30T00:18:26.660 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:26.660 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:26.662 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.660+0000 7f943e145700 10 mgrc shutdown 2023-09-30T00:18:26.662 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.660+0000 7f943e145700 1 -- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f94240987b0 msgr2=0x7f942409ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:26.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.660+0000 7f943e145700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f94240987b0 0x7f942409ac60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f943813d940 tx=0x7f9428008040 comp rx=0 tx=0).stop 2023-09-30T00:18:26.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.660+0000 7f943e145700 1 -- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 msgr2=0x7f943813bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:26.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.660+0000 7f943e145700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 0x7f943813bfe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f942c009f00 tx=0x7f942c017000 comp rx=0 tx=0).stop 2023-09-30T00:18:26.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.660+0000 7f943e145700 1 -- 172.21.15.94:0/2763623842 shutdown_connections 2023-09-30T00:18:26.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.660+0000 7f943e145700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f94240987b0 0x7f942409ac60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:26.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.661+0000 7f943e145700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9438070c50 0x7f943813bfe0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:26.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.661+0000 7f943e145700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f943813fd70 0x7f9438150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:26.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.661+0000 7f943e145700 1 --2- 172.21.15.94:0/2763623842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94380b7160 0x7f94381505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:26.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.661+0000 7f943e145700 1 -- 172.21.15.94:0/2763623842 >> 172.21.15.94:0/2763623842 conn(0x7f943806a090 msgr2=0x7f94380b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:26.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.661+0000 7f943e145700 1 -- 172.21.15.94:0/2763623842 shutdown_connections 2023-09-30T00:18:26.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:26.661+0000 7f943e145700 1 -- 172.21.15.94:0/2763623842 wait complete. 2023-09-30T00:18:29.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:29.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.799+0000 7f735b1ad700 1 Processor -- start 2023-09-30T00:18:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.799+0000 7f735b1ad700 1 -- start start 2023-09-30T00:18:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.799+0000 7f735b1ad700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f73540b7160 0x7f73540a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.799+0000 7f735b1ad700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f735413fd70 0x7f7354140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.799+0000 7f735b1ad700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73540a1310 con 0x7f7354070c50 2023-09-30T00:18:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.799+0000 7f735b1ad700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7354071020 con 0x7f735413fd70 2023-09-30T00:18:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.799+0000 7f735b1ad700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73540b76a0 con 0x7f73540b7160 2023-09-30T00:18:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.799+0000 7f735974a700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f735413fd70 0x7f7354140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.799+0000 7f735974a700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f735413fd70 0x7f7354140180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47402/0 (socket says 172.21.15.94:47402) 2023-09-30T00:18:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.799+0000 7f735974a700 1 -- 172.21.15.94:0/2604941634 learned_addr learned my addr 172.21.15.94:0/2604941634 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.800+0000 7f734bfff700 1 --2- 172.21.15.94:0/2604941634 >> v2:172.21.15.121:3300/0 conn(0x7f73540b7160 0x7f73540a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.800+0000 7f735974a700 1 -- 172.21.15.94:0/2604941634 >> v2:172.21.15.121:3300/0 conn(0x7f73540b7160 msgr2=0x7f73540a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.800+0000 7f735974a700 1 --2- 172.21.15.94:0/2604941634 >> v2:172.21.15.121:3300/0 conn(0x7f73540b7160 0x7f73540a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.800+0000 7f735974a700 1 -- 172.21.15.94:0/2604941634 >> v1:172.21.15.121:6789/0 conn(0x7f7354070c50 legacy=0x7f73540a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.800+0000 7f735974a700 1 -- 172.21.15.94:0/2604941634 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73540b7940 con 0x7f735413fd70 2023-09-30T00:18:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.800+0000 7f734bfff700 1 --2- 172.21.15.94:0/2604941634 >> v2:172.21.15.121:3300/0 conn(0x7f73540b7160 0x7f73540a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.800+0000 7f735974a700 1 --2- 172.21.15.94:0/2604941634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f735413fd70 0x7f7354140180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f735406a880 tx=0x7f734c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f01e2cbedbd765 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.800+0000 7f734b7fe700 1 -- 172.21.15.94:0/2604941634 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c00e070 con 0x7f735413fd70 2023-09-30T00:18:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.800+0000 7f734b7fe700 1 -- 172.21.15.94:0/2604941634 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c00bae0 con 0x7f735413fd70 2023-09-30T00:18:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.800+0000 7f734b7fe700 1 -- 172.21.15.94:0/2604941634 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c0092e0 con 0x7f735413fd70 2023-09-30T00:18:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.801+0000 7f735b1ad700 1 -- 172.21.15.94:0/2604941634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f735413fd70 msgr2=0x7f7354140180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.801+0000 7f735b1ad700 1 --2- 172.21.15.94:0/2604941634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f735413fd70 0x7f7354140180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f735406a880 tx=0x7f734c009840 comp rx=0 tx=0).stop 2023-09-30T00:18:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.801+0000 7f735b1ad700 1 -- 172.21.15.94:0/2604941634 shutdown_connections 2023-09-30T00:18:29.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.801+0000 7f735b1ad700 1 --2- 172.21.15.94:0/2604941634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f735413fd70 0x7f7354140180 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:29.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.801+0000 7f735b1ad700 1 --2- 172.21.15.94:0/2604941634 >> v2:172.21.15.121:3300/0 conn(0x7f73540b7160 0x7f73540a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:29.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.801+0000 7f735b1ad700 1 -- 172.21.15.94:0/2604941634 >> 172.21.15.94:0/2604941634 conn(0x7f735406a090 msgr2=0x7f73540702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:29.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.801+0000 7f735b1ad700 1 -- 172.21.15.94:0/2604941634 shutdown_connections 2023-09-30T00:18:29.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.801+0000 7f735b1ad700 1 -- 172.21.15.94:0/2604941634 wait complete. 2023-09-30T00:18:29.810 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.802+0000 7f735b1ad700 1 Processor -- start 2023-09-30T00:18:29.810 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.802+0000 7f735b1ad700 1 -- start start 2023-09-30T00:18:29.810 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.802+0000 7f735b1ad700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7354070c50 0x7f73541448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:29.810 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.802+0000 7f735b1ad700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73540b7160 0x7f735413aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:29.810 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.802+0000 7f735b1ad700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f735413fd70 0x7f73541432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:29.811 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.802+0000 7f735b1ad700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735413b110 con 0x7f7354070c50 2023-09-30T00:18:29.811 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.802+0000 7f735b1ad700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735413b3d0 con 0x7f73540b7160 2023-09-30T00:18:29.811 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.802+0000 7f735b1ad700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73540a8f80 con 0x7f735413fd70 2023-09-30T00:18:29.811 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.802+0000 7f7358f49700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7354070c50 0x7f73541448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:29.811 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.802+0000 7f735974a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f735413fd70 0x7f73541432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:29.811 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f735974a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f735413fd70 0x7f73541432d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44840/0 (socket says 172.21.15.94:44840) 2023-09-30T00:18:29.811 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f7358f49700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7354070c50 0x7f73541448e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47404/0 (socket says 172.21.15.94:47404) 2023-09-30T00:18:29.811 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f735974a700 1 -- 172.21.15.94:0/3800012265 learned_addr learned my addr 172.21.15.94:0/3800012265 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:29.811 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f734bfff700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73540b7160 0x7f735413aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:29.811 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f735974a700 1 -- 172.21.15.94:0/3800012265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73540b7160 msgr2=0x7f735413aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:29.811 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f735974a700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73540b7160 0x7f735413aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f735974a700 1 -- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7354070c50 msgr2=0x7f73541448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f735974a700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7354070c50 0x7f73541448e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f735974a700 1 -- 172.21.15.94:0/3800012265 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c009ae0 con 0x7f735413fd70 2023-09-30T00:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f734bfff700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73540b7160 0x7f735413aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f7358f49700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7354070c50 0x7f73541448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f735974a700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f735413fd70 0x7f73541432d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f734c009880 tx=0x7f734c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f7348ff9700 1 -- 172.21.15.94:0/3800012265 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c0032f0 con 0x7f735413fd70 2023-09-30T00:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f7348ff9700 1 -- 172.21.15.94:0/3800012265 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c008dd0 con 0x7f735413fd70 2023-09-30T00:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f735b1ad700 1 -- 172.21.15.94:0/3800012265 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7354143810 con 0x7f735413fd70 2023-09-30T00:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f7348ff9700 1 -- 172.21.15.94:0/3800012265 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c01c640 con 0x7f735413fd70 2023-09-30T00:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.803+0000 7f735b1ad700 1 -- 172.21.15.94:0/3800012265 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73540a3c40 con 0x7f735413fd70 2023-09-30T00:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.804+0000 7f735b1ad700 1 -- 172.21.15.94:0/3800012265 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735413c0a0 con 0x7f735413fd70 2023-09-30T00:18:29.813 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.805+0000 7f7348ff9700 1 -- 172.21.15.94:0/3800012265 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f734c003040 con 0x7f735413fd70 2023-09-30T00:18:29.813 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.805+0000 7f7348ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:29.813 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.805+0000 7f7348ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:29.813 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.805+0000 7f7348ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:29.813 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.805+0000 7f7348ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:29.813 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.805+0000 7f7348ff9700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7340098700 0x7f734009abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:29.813 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.806+0000 7f7358f49700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7340098700 0x7f734009abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:29.813 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.806+0000 7f7348ff9700 1 -- 172.21.15.94:0/3800012265 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f734c0c1c70 con 0x7f735413fd70 2023-09-30T00:18:29.813 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.806+0000 7f7358f49700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7340098700 0x7f734009abb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7350001dc0 tx=0x7f7350009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:29.814 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:29.810+0000 7f7348ff9700 1 -- 172.21.15.94:0/3800012265 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f734c08f3b0 con 0x7f735413fd70 2023-09-30T00:18:30.033 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.032+0000 7f735b1ad700 1 -- 172.21.15.94:0/3800012265 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7354071630 con 0x7f735413fd70 2023-09-30T00:18:30.034 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.032+0000 7f7348ff9700 1 -- 172.21.15.94:0/3800012265 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f734c0929d0 con 0x7f735413fd70 2023-09-30T00:18:30.034 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:30.034 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:30.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 10 mgrc shutdown 2023-09-30T00:18:30.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 1 -- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7340098700 msgr2=0x7f734009abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:30.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7340098700 0x7f734009abb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7350001dc0 tx=0x7f7350009040 comp rx=0 tx=0).stop 2023-09-30T00:18:30.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 1 -- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f735413fd70 msgr2=0x7f73541432d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:30.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f735413fd70 0x7f73541432d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f734c009880 tx=0x7f734c018000 comp rx=0 tx=0).stop 2023-09-30T00:18:30.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 1 -- 172.21.15.94:0/3800012265 shutdown_connections 2023-09-30T00:18:30.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7340098700 0x7f734009abb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:30.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f735413fd70 0x7f73541432d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:30.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7354070c50 0x7f73541448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:30.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 1 --2- 172.21.15.94:0/3800012265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73540b7160 0x7f735413aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:30.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 1 -- 172.21.15.94:0/3800012265 >> 172.21.15.94:0/3800012265 conn(0x7f735406a090 msgr2=0x7f7354067cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:30.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 1 -- 172.21.15.94:0/3800012265 shutdown_connections 2023-09-30T00:18:30.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:30.034+0000 7f735b1ad700 1 -- 172.21.15.94:0/3800012265 wait complete. 2023-09-30T00:18:33.050 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:33.176 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.174+0000 7f614b117700 1 Processor -- start 2023-09-30T00:18:33.176 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.174+0000 7f614b117700 1 -- start start 2023-09-30T00:18:33.178 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f614b117700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f614406aaa0 0x7f61440b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:33.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f614b117700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f614409ca70 0x7f614409ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:33.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f614b117700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614406ae70 con 0x7f61440a3ff0 2023-09-30T00:18:33.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f614b117700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61440b40e0 con 0x7f614406aaa0 2023-09-30T00:18:33.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f614b117700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61440b48c0 con 0x7f614409ca70 2023-09-30T00:18:33.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f6148eb3700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f614406aaa0 0x7f61440b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:33.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f61496b4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f614409ca70 0x7f614409ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:33.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f6143fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f61440a3ff0 0x7f61440b24b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57820/0 (socket says 172.21.15.94:57820) 2023-09-30T00:18:33.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f6148eb3700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f614406aaa0 0x7f61440b3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47418/0 (socket says 172.21.15.94:47418) 2023-09-30T00:18:33.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f61496b4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f614409ca70 0x7f614409ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44852/0 (socket says 172.21.15.94:44852) 2023-09-30T00:18:33.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f6143fff700 1 -- 172.21.15.94:0/3639255392 learned_addr learned my addr 172.21.15.94:0/3639255392 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:33.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f61496b4700 1 -- 172.21.15.94:0/3639255392 >> v2:172.21.15.94:3300/0 conn(0x7f614406aaa0 msgr2=0x7f61440b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f61496b4700 1 --2- 172.21.15.94:0/3639255392 >> v2:172.21.15.94:3300/0 conn(0x7f614406aaa0 0x7f61440b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f61496b4700 1 -- 172.21.15.94:0/3639255392 >> v1:172.21.15.94:6789/0 conn(0x7f61440a3ff0 legacy=0x7f61440b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f61496b4700 1 -- 172.21.15.94:0/3639255392 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61440a45c0 con 0x7f614409ca70 2023-09-30T00:18:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f6148eb3700 1 --2- 172.21.15.94:0/3639255392 >> v2:172.21.15.94:3300/0 conn(0x7f614406aaa0 0x7f61440b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.175+0000 7f61496b4700 1 --2- 172.21.15.94:0/3639255392 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f614409ca70 0x7f614409ce90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f613c00d0d0 tx=0x7f613c00b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db0b003e72517ebd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.176+0000 7f61437fe700 1 -- 172.21.15.94:0/3639255392 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c010070 con 0x7f614409ca70 2023-09-30T00:18:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.176+0000 7f61437fe700 1 -- 172.21.15.94:0/3639255392 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c00b2e0 con 0x7f614409ca70 2023-09-30T00:18:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.176+0000 7f61437fe700 1 -- 172.21.15.94:0/3639255392 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c00dd80 con 0x7f614409ca70 2023-09-30T00:18:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.176+0000 7f614b117700 1 -- 172.21.15.94:0/3639255392 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f614409ca70 msgr2=0x7f614409ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.176+0000 7f614b117700 1 --2- 172.21.15.94:0/3639255392 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f614409ca70 0x7f614409ce90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f613c00d0d0 tx=0x7f613c00b990 comp rx=0 tx=0).stop 2023-09-30T00:18:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.176+0000 7f614b117700 1 -- 172.21.15.94:0/3639255392 shutdown_connections 2023-09-30T00:18:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.176+0000 7f614b117700 1 --2- 172.21.15.94:0/3639255392 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f614409ca70 0x7f614409ce90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:33.181 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.176+0000 7f614b117700 1 --2- 172.21.15.94:0/3639255392 >> v2:172.21.15.94:3300/0 conn(0x7f614406aaa0 0x7f61440b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:33.181 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.176+0000 7f614b117700 1 -- 172.21.15.94:0/3639255392 >> 172.21.15.94:0/3639255392 conn(0x7f6144068730 msgr2=0x7f6144068b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:33.181 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.176+0000 7f614b117700 1 -- 172.21.15.94:0/3639255392 shutdown_connections 2023-09-30T00:18:33.181 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.176+0000 7f614b117700 1 -- 172.21.15.94:0/3639255392 wait complete. 2023-09-30T00:18:33.181 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.177+0000 7f614b117700 1 Processor -- start 2023-09-30T00:18:33.181 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.177+0000 7f614b117700 1 -- start start 2023-09-30T00:18:33.183 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f614b117700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f614406aaa0 0x7f61440a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:33.183 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f614b117700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f614409ca70 0x7f614414f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:33.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f614b117700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f61440a3ff0 0x7f614414f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:33.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f614b117700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614414fc80 con 0x7f614406aaa0 2023-09-30T00:18:33.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f614b117700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6144150150 con 0x7f614409ca70 2023-09-30T00:18:33.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f614b117700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614414fe90 con 0x7f61440a3ff0 2023-09-30T00:18:33.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f6148eb3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f614406aaa0 0x7f61440a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:33.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f61496b4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f61440a3ff0 0x7f614414f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:33.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f6148eb3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f614406aaa0 0x7f61440a1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47420/0 (socket says 172.21.15.94:47420) 2023-09-30T00:18:33.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f6148eb3700 1 -- 172.21.15.94:0/1525815924 learned_addr learned my addr 172.21.15.94:0/1525815924 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:33.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f61496b4700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f61440a3ff0 0x7f614414f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44856/0 (socket says 172.21.15.94:44856) 2023-09-30T00:18:33.184 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f6143fff700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f614409ca70 0x7f614414f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f6148eb3700 1 -- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f61440a3ff0 msgr2=0x7f614414f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f6148eb3700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f61440a3ff0 0x7f614414f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f6148eb3700 1 -- 172.21.15.94:0/1525815924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f614409ca70 msgr2=0x7f614414f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f6148eb3700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f614409ca70 0x7f614414f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f6148eb3700 1 -- 172.21.15.94:0/1525815924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c009420 con 0x7f614406aaa0 2023-09-30T00:18:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f6143fff700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f614409ca70 0x7f614414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:18:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f6148eb3700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f614406aaa0 0x7f61440a1a80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f612c00ec80 tx=0x7f612c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.178+0000 7f6140ff9700 1 -- 172.21.15.94:0/1525815924 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c00c920 con 0x7f614406aaa0 2023-09-30T00:18:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.179+0000 7f6140ff9700 1 -- 172.21.15.94:0/1525815924 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612c00ca80 con 0x7f614406aaa0 2023-09-30T00:18:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.179+0000 7f614b117700 1 -- 172.21.15.94:0/1525815924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613c00bc30 con 0x7f614406aaa0 2023-09-30T00:18:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.179+0000 7f6140ff9700 1 -- 172.21.15.94:0/1525815924 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c00ce00 con 0x7f614406aaa0 2023-09-30T00:18:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.179+0000 7f614b117700 1 -- 172.21.15.94:0/1525815924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61440a3310 con 0x7f614406aaa0 2023-09-30T00:18:33.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.180+0000 7f614b117700 1 -- 172.21.15.94:0/1525815924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614406fbd0 con 0x7f614406aaa0 2023-09-30T00:18:33.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.181+0000 7f6140ff9700 1 -- 172.21.15.94:0/1525815924 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f612c013680 con 0x7f614406aaa0 2023-09-30T00:18:33.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.181+0000 7f6140ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:33.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.181+0000 7f6140ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:33.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.181+0000 7f6140ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:33.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.181+0000 7f6140ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:33.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.181+0000 7f6140ff9700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6124098780 0x7f612409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:33.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.181+0000 7f6143fff700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6124098780 0x7f612409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:33.186 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.181+0000 7f6140ff9700 1 -- 172.21.15.94:0/1525815924 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f612c0c0f70 con 0x7f614406aaa0 2023-09-30T00:18:33.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.181+0000 7f6143fff700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6124098780 0x7f612409ac30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f61440a39c0 tx=0x7f6134009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:33.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.185+0000 7f6140ff9700 1 -- 172.21.15.94:0/1525815924 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f612c08e620 con 0x7f614406aaa0 2023-09-30T00:18:33.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.408+0000 7f614b117700 1 -- 172.21.15.94:0/1525815924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f61440b4360 con 0x7f614406aaa0 2023-09-30T00:18:33.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.408+0000 7f6140ff9700 1 -- 172.21.15.94:0/1525815924 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f612c021100 con 0x7f614406aaa0 2023-09-30T00:18:33.410 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:33.411 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:33.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 10 mgrc shutdown 2023-09-30T00:18:33.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 1 -- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6124098780 msgr2=0x7f612409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:33.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6124098780 0x7f612409ac30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f61440a39c0 tx=0x7f6134009040 comp rx=0 tx=0).stop 2023-09-30T00:18:33.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 1 -- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f614406aaa0 msgr2=0x7f61440a1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:33.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f614406aaa0 0x7f61440a1a80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f612c00ec80 tx=0x7f612c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:33.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 1 -- 172.21.15.94:0/1525815924 shutdown_connections 2023-09-30T00:18:33.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6124098780 0x7f612409ac30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:33.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f61440a3ff0 0x7f614414f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:33.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f614406aaa0 0x7f61440a1a80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:33.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 1 --2- 172.21.15.94:0/1525815924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f614409ca70 0x7f614414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:33.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 1 -- 172.21.15.94:0/1525815924 >> 172.21.15.94:0/1525815924 conn(0x7f6144068730 msgr2=0x7f6144068b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:33.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 1 -- 172.21.15.94:0/1525815924 shutdown_connections 2023-09-30T00:18:33.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:33.410+0000 7f614b117700 1 -- 172.21.15.94:0/1525815924 wait complete. 2023-09-30T00:18:36.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:36.552 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.548+0000 7f1cfdd2f700 1 Processor -- start 2023-09-30T00:18:36.552 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.548+0000 7f1cfdd2f700 1 -- start start 2023-09-30T00:18:36.552 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.549+0000 7f1cfdd2f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 0x7f1cf80a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:36.552 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.549+0000 7f1cfdd2f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf80a1310 con 0x7f1cf80b7160 2023-09-30T00:18:36.552 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.549+0000 7f1cfdd2f700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf8071020 con 0x7f1cf813fd70 2023-09-30T00:18:36.552 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.549+0000 7f1cfdd2f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf80b76a0 con 0x7f1cf8070c50 2023-09-30T00:18:36.552 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.549+0000 7f1cf77fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 0x7f1cf80a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.549+0000 7f1cf6ffd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1cf80b7160 0x7f1cf80a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36626/0 (socket says 172.21.15.94:36626) 2023-09-30T00:18:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.549+0000 7f1cf6ffd700 1 -- 172.21.15.94:0/4220176772 learned_addr learned my addr 172.21.15.94:0/4220176772 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.549+0000 7f1cf67fc700 1 -- 172.21.15.94:0/4220176772 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637942626 0 0) 0x7f1cec001110 con 0x7f1cf80b7160 2023-09-30T00:18:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.549+0000 7f1cf77fe700 1 -- 172.21.15.94:0/4220176772 >> v1:172.21.15.121:6789/0 conn(0x7f1cf813fd70 legacy=0x7f1cf814b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.549+0000 7f1cf77fe700 1 -- 172.21.15.94:0/4220176772 >> v1:172.21.15.94:6789/0 conn(0x7f1cf80b7160 legacy=0x7f1cf80a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.549+0000 7f1cf77fe700 1 -- 172.21.15.94:0/4220176772 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf80b7940 con 0x7f1cf8070c50 2023-09-30T00:18:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.550+0000 7f1cf77fe700 1 --2- 172.21.15.94:0/4220176772 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 0x7f1cf80a0dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009e90 tx=0x7f1ce8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39a0a3b55c35f160 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.550+0000 7f1cf67fc700 1 -- 172.21.15.94:0/4220176772 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce800e070 con 0x7f1cf8070c50 2023-09-30T00:18:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.550+0000 7f1cf67fc700 1 -- 172.21.15.94:0/4220176772 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce800bc50 con 0x7f1cf8070c50 2023-09-30T00:18:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.550+0000 7f1cf67fc700 1 -- 172.21.15.94:0/4220176772 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8012430 con 0x7f1cf8070c50 2023-09-30T00:18:36.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.550+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/4220176772 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 msgr2=0x7f1cf80a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:36.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.550+0000 7f1cfdd2f700 1 --2- 172.21.15.94:0/4220176772 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 0x7f1cf80a0dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009e90 tx=0x7f1ce8009780 comp rx=0 tx=0).stop 2023-09-30T00:18:36.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.550+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/4220176772 shutdown_connections 2023-09-30T00:18:36.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.550+0000 7f1cfdd2f700 1 --2- 172.21.15.94:0/4220176772 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 0x7f1cf80a0dd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:36.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.550+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/4220176772 >> 172.21.15.94:0/4220176772 conn(0x7f1cf806a090 msgr2=0x7f1cf80702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:36.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.550+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/4220176772 shutdown_connections 2023-09-30T00:18:36.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.550+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/4220176772 wait complete. 2023-09-30T00:18:36.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.551+0000 7f1cfdd2f700 1 Processor -- start 2023-09-30T00:18:36.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.551+0000 7f1cfdd2f700 1 -- start start 2023-09-30T00:18:36.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cfdd2f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 0x7f1cf80a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:36.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cfdd2f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cf80b7160 0x7f1cf80ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:36.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cfdd2f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cf813fd70 0x7f1cf80aac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:36.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cfdd2f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf80ab930 con 0x7f1cf80b7160 2023-09-30T00:18:36.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cfdd2f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf80acb20 con 0x7f1cf813fd70 2023-09-30T00:18:36.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cfdd2f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf81447a0 con 0x7f1cf8070c50 2023-09-30T00:18:36.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cf77fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 0x7f1cf80a7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:36.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cf6ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cf80b7160 0x7f1cf80ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:36.558 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cf77fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 0x7f1cf80a7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:36882/0 (socket says 172.21.15.94:36882) 2023-09-30T00:18:36.558 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cf77fe700 1 -- 172.21.15.94:0/414411629 learned_addr learned my addr 172.21.15.94:0/414411629 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:36.558 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cf7fff700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cf813fd70 0x7f1cf80aac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:36.558 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cf77fe700 1 -- 172.21.15.94:0/414411629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cf813fd70 msgr2=0x7f1cf80aac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:36.558 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cf77fe700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cf813fd70 0x7f1cf80aac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:36.558 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cf77fe700 1 -- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cf80b7160 msgr2=0x7f1cf80ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:36.558 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cf77fe700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cf80b7160 0x7f1cf80ac5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:36.558 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cf77fe700 1 -- 172.21.15.94:0/414411629 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce8009a20 con 0x7f1cf8070c50 2023-09-30T00:18:36.559 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.552+0000 7f1cf6ffd700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cf80b7160 0x7f1cf80ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:36.559 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.553+0000 7f1cf77fe700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 0x7f1cf80a7390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80a97e0 tx=0x7f1ce8018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:36.559 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.553+0000 7f1cdf7fe700 1 -- 172.21.15.94:0/414411629 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8012df0 con 0x7f1cf8070c50 2023-09-30T00:18:36.559 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.553+0000 7f1cdf7fe700 1 -- 172.21.15.94:0/414411629 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce8021070 con 0x7f1cf8070c50 2023-09-30T00:18:36.559 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.553+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/414411629 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf80ab670 con 0x7f1cf8070c50 2023-09-30T00:18:36.559 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.553+0000 7f1cdf7fe700 1 -- 172.21.15.94:0/414411629 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8026680 con 0x7f1cf8070c50 2023-09-30T00:18:36.559 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.553+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/414411629 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf80a7a70 con 0x7f1cf8070c50 2023-09-30T00:18:36.559 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.554+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/414411629 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf8071630 con 0x7f1cf8070c50 2023-09-30T00:18:36.559 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.555+0000 7f1cdf7fe700 1 -- 172.21.15.94:0/414411629 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f1ce8023030 con 0x7f1cf8070c50 2023-09-30T00:18:36.559 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.555+0000 7f1cdf7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:36.559 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.555+0000 7f1cdf7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:36.559 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.555+0000 7f1cdf7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:36.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.555+0000 7f1cdf7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:36.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.555+0000 7f1cdf7fe700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1ce4098780 0x7f1ce409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:36.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.555+0000 7f1cf6ffd700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1ce4098780 0x7f1ce409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:36.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.555+0000 7f1cdf7fe700 1 -- 172.21.15.94:0/414411629 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f1ce80c2360 con 0x7f1cf8070c50 2023-09-30T00:18:36.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.556+0000 7f1cf6ffd700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1ce4098780 0x7f1ce409ac30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1cf813d120 tx=0x7f1cec007b90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:36.560 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.559+0000 7f1cdf7fe700 1 -- 172.21.15.94:0/414411629 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ce808faa0 con 0x7f1cf8070c50 2023-09-30T00:18:36.782 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.781+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/414411629 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1cf813f620 con 0x7f1cf8070c50 2023-09-30T00:18:36.783 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.781+0000 7f1cdf7fe700 1 -- 172.21.15.94:0/414411629 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1ce80930c0 con 0x7f1cf8070c50 2023-09-30T00:18:36.783 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:36.783 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:36.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 10 mgrc shutdown 2023-09-30T00:18:36.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1ce4098780 msgr2=0x7f1ce409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:36.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1ce4098780 0x7f1ce409ac30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1cf813d120 tx=0x7f1cec007b90 comp rx=0 tx=0).stop 2023-09-30T00:18:36.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 msgr2=0x7f1cf80a7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:36.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 0x7f1cf80a7390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80a97e0 tx=0x7f1ce8018000 comp rx=0 tx=0).stop 2023-09-30T00:18:36.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/414411629 shutdown_connections 2023-09-30T00:18:36.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1ce4098780 0x7f1ce409ac30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:36.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1cf8070c50 0x7f1cf80a7390 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:36.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cf813fd70 0x7f1cf80aac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:36.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 1 --2- 172.21.15.94:0/414411629 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cf80b7160 0x7f1cf80ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:36.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/414411629 >> 172.21.15.94:0/414411629 conn(0x7f1cf806a090 msgr2=0x7f1cf80a8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:36.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/414411629 shutdown_connections 2023-09-30T00:18:36.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:36.783+0000 7f1cfdd2f700 1 -- 172.21.15.94:0/414411629 wait complete. 2023-09-30T00:18:39.062 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:18:39.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:18:39.798 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:39.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.922+0000 7f2827dcc700 1 Processor -- start 2023-09-30T00:18:39.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.922+0000 7f2827dcc700 1 -- start start 2023-09-30T00:18:39.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2827dcc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f28200691f0 0x7f2820067620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:39.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2827dcc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2820068a70 0x7f28200a26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:39.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2827dcc700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2820070910 con 0x7f2820070540 2023-09-30T00:18:39.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2827dcc700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2820067e20 con 0x7f28200691f0 2023-09-30T00:18:39.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2827dcc700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28200680e0 con 0x7f2820068a70 2023-09-30T00:18:39.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2826369700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2820068a70 0x7f28200a26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2825b68700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2820070540 0x7f28200a0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36630/0 (socket says 172.21.15.94:36630) 2023-09-30T00:18:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2825b68700 1 -- 172.21.15.94:0/2388467850 learned_addr learned my addr 172.21.15.94:0/2388467850 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2825367700 1 --2- 172.21.15.94:0/2388467850 >> v2:172.21.15.121:3300/0 conn(0x7f28200691f0 0x7f2820067620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2826369700 1 -- 172.21.15.94:0/2388467850 >> v2:172.21.15.121:3300/0 conn(0x7f28200691f0 msgr2=0x7f2820067620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2824b66700 1 -- 172.21.15.94:0/2388467850 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737783742 0 0) 0x7f2820070910 con 0x7f2820070540 2023-09-30T00:18:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2826369700 1 --2- 172.21.15.94:0/2388467850 >> v2:172.21.15.121:3300/0 conn(0x7f28200691f0 0x7f2820067620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2826369700 1 -- 172.21.15.94:0/2388467850 >> v1:172.21.15.94:6789/0 conn(0x7f2820070540 legacy=0x7f28200a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.923+0000 7f2826369700 1 -- 172.21.15.94:0/2388467850 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2820067b60 con 0x7f2820068a70 2023-09-30T00:18:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.924+0000 7f2826369700 1 --2- 172.21.15.94:0/2388467850 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2820068a70 0x7f28200a26f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f28180098e0 tx=0x7f28180091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e45d923de4f874b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.924+0000 7f2824b66700 1 -- 172.21.15.94:0/2388467850 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818009e80 con 0x7f2820068a70 2023-09-30T00:18:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.924+0000 7f2824b66700 1 -- 172.21.15.94:0/2388467850 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281800b860 con 0x7f2820068a70 2023-09-30T00:18:39.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.924+0000 7f2824b66700 1 -- 172.21.15.94:0/2388467850 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281800bbe0 con 0x7f2820068a70 2023-09-30T00:18:39.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.924+0000 7f2827dcc700 1 -- 172.21.15.94:0/2388467850 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2820068a70 msgr2=0x7f28200a26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:39.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.924+0000 7f2827dcc700 1 --2- 172.21.15.94:0/2388467850 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2820068a70 0x7f28200a26f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f28180098e0 tx=0x7f28180091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:39.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.924+0000 7f2827dcc700 1 -- 172.21.15.94:0/2388467850 shutdown_connections 2023-09-30T00:18:39.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.924+0000 7f2827dcc700 1 --2- 172.21.15.94:0/2388467850 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2820068a70 0x7f28200a26f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:39.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.924+0000 7f2827dcc700 1 --2- 172.21.15.94:0/2388467850 >> v2:172.21.15.121:3300/0 conn(0x7f28200691f0 0x7f2820067620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:39.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.924+0000 7f2827dcc700 1 -- 172.21.15.94:0/2388467850 >> 172.21.15.94:0/2388467850 conn(0x7f28200a57c0 msgr2=0x7f28200a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:39.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.924+0000 7f2827dcc700 1 -- 172.21.15.94:0/2388467850 shutdown_connections 2023-09-30T00:18:39.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.924+0000 7f2827dcc700 1 -- 172.21.15.94:0/2388467850 wait complete. 2023-09-30T00:18:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.925+0000 7f2827dcc700 1 Processor -- start 2023-09-30T00:18:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.925+0000 7f2827dcc700 1 -- start start 2023-09-30T00:18:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2827dcc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2820068a70 0x7f28200988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2827dcc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28200691f0 0x7f282013e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2827dcc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2820070540 0x7f282013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2827dcc700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282006f3d0 con 0x7f28200691f0 2023-09-30T00:18:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2827dcc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282013f280 con 0x7f2820070540 2023-09-30T00:18:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2827dcc700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282013f490 con 0x7f2820068a70 2023-09-30T00:18:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2825367700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28200691f0 0x7f282013e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:39.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2825b68700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2820068a70 0x7f28200988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:39.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2825367700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28200691f0 0x7f282013e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57516/0 (socket says 172.21.15.94:57516) 2023-09-30T00:18:39.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2825367700 1 -- 172.21.15.94:0/3808033510 learned_addr learned my addr 172.21.15.94:0/3808033510 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:39.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2826369700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2820070540 0x7f282013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:39.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2825367700 1 -- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2820068a70 msgr2=0x7f28200988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:39.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2825367700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2820068a70 0x7f28200988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:39.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2825367700 1 -- 172.21.15.94:0/3808033510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2820070540 msgr2=0x7f282013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:39.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2825367700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2820070540 0x7f282013ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:39.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2825367700 1 -- 172.21.15.94:0/3808033510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2818009420 con 0x7f28200691f0 2023-09-30T00:18:39.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2826369700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2820070540 0x7f282013ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:18:39.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2825367700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28200691f0 0x7f282013e800 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f281000d860 tx=0x7f2810008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:39.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2815ffb700 1 -- 172.21.15.94:0/3808033510 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2810018070 con 0x7f28200691f0 2023-09-30T00:18:39.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2815ffb700 1 -- 172.21.15.94:0/3808033510 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28100125f0 con 0x7f28200691f0 2023-09-30T00:18:39.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2815ffb700 1 -- 172.21.15.94:0/3808033510 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2810012990 con 0x7f28200691f0 2023-09-30T00:18:39.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.926+0000 7f2827dcc700 1 -- 172.21.15.94:0/3808033510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282013f7b0 con 0x7f28200691f0 2023-09-30T00:18:39.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.927+0000 7f2827dcc700 1 -- 172.21.15.94:0/3808033510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282013da90 con 0x7f28200691f0 2023-09-30T00:18:39.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.928+0000 7f2827dcc700 1 -- 172.21.15.94:0/3808033510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f282006a6a0 con 0x7f28200691f0 2023-09-30T00:18:39.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.928+0000 7f2815ffb700 1 -- 172.21.15.94:0/3808033510 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2810012d00 con 0x7f28200691f0 2023-09-30T00:18:39.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.928+0000 7f2815ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:39.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.929+0000 7f2815ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:39.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.929+0000 7f2815ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:39.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.929+0000 7f2815ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:39.933 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.929+0000 7f2815ffb700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2804098780 0x7f280409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:39.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.929+0000 7f2825b68700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2804098780 0x7f280409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:39.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.929+0000 7f2815ffb700 1 -- 172.21.15.94:0/3808033510 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4330+0+0 (secure 0 0 0) 0x7f28100c0650 con 0x7f28200691f0 2023-09-30T00:18:39.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.929+0000 7f2825b68700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2804098780 0x7f280409ac30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f281c0030c0 tx=0x7f281c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:39.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:39.933+0000 7f2815ffb700 1 -- 172.21.15.94:0/3808033510 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f281008dd90 con 0x7f28200691f0 2023-09-30T00:18:40.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.155+0000 7f2827dcc700 1 -- 172.21.15.94:0/3808033510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f282006fbd0 con 0x7f28200691f0 2023-09-30T00:18:40.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.155+0000 7f2815ffb700 1 -- 172.21.15.94:0/3808033510 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f28100913b0 con 0x7f28200691f0 2023-09-30T00:18:40.157 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:40.157 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:40.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 10 mgrc shutdown 2023-09-30T00:18:40.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 1 -- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2804098780 msgr2=0x7f280409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:40.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2804098780 0x7f280409ac30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f281c0030c0 tx=0x7f281c009040 comp rx=0 tx=0).stop 2023-09-30T00:18:40.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 1 -- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28200691f0 msgr2=0x7f282013e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:40.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28200691f0 0x7f282013e800 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f281000d860 tx=0x7f2810008040 comp rx=0 tx=0).stop 2023-09-30T00:18:40.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 1 -- 172.21.15.94:0/3808033510 shutdown_connections 2023-09-30T00:18:40.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2804098780 0x7f280409ac30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:40.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2820068a70 0x7f28200988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:40.160 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2820070540 0x7f282013ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:40.160 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 1 --2- 172.21.15.94:0/3808033510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28200691f0 0x7f282013e800 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:40.160 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 1 -- 172.21.15.94:0/3808033510 >> 172.21.15.94:0/3808033510 conn(0x7f28200a57c0 msgr2=0x7f28200630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:40.160 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 1 -- 172.21.15.94:0/3808033510 shutdown_connections 2023-09-30T00:18:40.160 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:40.157+0000 7f2827dcc700 1 -- 172.21.15.94:0/3808033510 wait complete. 2023-09-30T00:18:43.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:43.295 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f6134948700 1 Processor -- start 2023-09-30T00:18:43.295 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f6134948700 1 -- start start 2023-09-30T00:18:43.295 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f6134948700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6130070c50 0x7f61300a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:43.295 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f6134948700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61300a1310 con 0x7f613013fd70 2023-09-30T00:18:43.295 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f6134948700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6130071020 con 0x7f61300b7160 2023-09-30T00:18:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f6134948700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61300b76a0 con 0x7f6130070c50 2023-09-30T00:18:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f612e59c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6130070c50 0x7f61300a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f612ed9d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f613013fd70 0x7f613014b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36634/0 (socket says 172.21.15.94:36634) 2023-09-30T00:18:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f612e59c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6130070c50 0x7f61300a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57522/0 (socket says 172.21.15.94:57522) 2023-09-30T00:18:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f612ed9d700 1 -- 172.21.15.94:0/2661801002 learned_addr learned my addr 172.21.15.94:0/2661801002 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f612e59c700 1 -- 172.21.15.94:0/2661801002 >> v1:172.21.15.121:6789/0 conn(0x7f61300b7160 legacy=0x7f61300a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f612e59c700 1 -- 172.21.15.94:0/2661801002 >> v1:172.21.15.94:6789/0 conn(0x7f613013fd70 legacy=0x7f613014b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f612e59c700 1 -- 172.21.15.94:0/2661801002 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61300b7940 con 0x7f6130070c50 2023-09-30T00:18:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.293+0000 7f612dd9b700 1 -- 172.21.15.94:0/2661801002 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211558348 0 0) 0x7f6128001490 con 0x7f613013fd70 2023-09-30T00:18:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.294+0000 7f612e59c700 1 --2- 172.21.15.94:0/2661801002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130070c50 0x7f61300a0dd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f61200098e0 tx=0x7f61200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3dd956c4196d5b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.294+0000 7f612dd9b700 1 -- 172.21.15.94:0/2661801002 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6120009e80 con 0x7f6130070c50 2023-09-30T00:18:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.294+0000 7f612dd9b700 1 -- 172.21.15.94:0/2661801002 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612000b8f0 con 0x7f6130070c50 2023-09-30T00:18:43.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.294+0000 7f612dd9b700 1 -- 172.21.15.94:0/2661801002 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612000bc70 con 0x7f6130070c50 2023-09-30T00:18:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.294+0000 7f6134948700 1 -- 172.21.15.94:0/2661801002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130070c50 msgr2=0x7f61300a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.294+0000 7f6134948700 1 --2- 172.21.15.94:0/2661801002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130070c50 0x7f61300a0dd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f61200098e0 tx=0x7f61200091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.294+0000 7f6134948700 1 -- 172.21.15.94:0/2661801002 shutdown_connections 2023-09-30T00:18:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.294+0000 7f6134948700 1 --2- 172.21.15.94:0/2661801002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6130070c50 0x7f61300a0dd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.294+0000 7f6134948700 1 -- 172.21.15.94:0/2661801002 >> 172.21.15.94:0/2661801002 conn(0x7f613006a090 msgr2=0x7f61300702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.294+0000 7f6134948700 1 -- 172.21.15.94:0/2661801002 shutdown_connections 2023-09-30T00:18:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.294+0000 7f6134948700 1 -- 172.21.15.94:0/2661801002 wait complete. 2023-09-30T00:18:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.295+0000 7f6134948700 1 Processor -- start 2023-09-30T00:18:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.295+0000 7f6134948700 1 -- start start 2023-09-30T00:18:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f6134948700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6130070c50 0x7f61300ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f6134948700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61300b7160 0x7f61300ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f6134948700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f613013fd70 0x7f6130153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f6134948700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61300ade60 con 0x7f61300b7160 2023-09-30T00:18:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f6134948700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6130144c70 con 0x7f613013fd70 2023-09-30T00:18:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f6134948700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6130151310 con 0x7f6130070c50 2023-09-30T00:18:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f612e59c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6130070c50 0x7f61300ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f612e59c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6130070c50 0x7f61300ad920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:36914/0 (socket says 172.21.15.94:36914) 2023-09-30T00:18:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f612e59c700 1 -- 172.21.15.94:0/336720668 learned_addr learned my addr 172.21.15.94:0/336720668 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f6125d9b700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61300b7160 0x7f61300ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f612ed9d700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f613013fd70 0x7f6130153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f612e59c700 1 -- 172.21.15.94:0/336720668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f613013fd70 msgr2=0x7f6130153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f612e59c700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f613013fd70 0x7f6130153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f612e59c700 1 -- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61300b7160 msgr2=0x7f61300ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f612e59c700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61300b7160 0x7f61300ac5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f612e59c700 1 -- 172.21.15.94:0/336720668 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6120009420 con 0x7f6130070c50 2023-09-30T00:18:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f6125d9b700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61300b7160 0x7f61300ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f612e59c700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6130070c50 0x7f61300ad920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f61300a97e0 tx=0x7f6120016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:43.302 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.296+0000 7f6126ffd700 1 -- 172.21.15.94:0/336720668 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612000bb30 con 0x7f6130070c50 2023-09-30T00:18:43.302 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.297+0000 7f6126ffd700 1 -- 172.21.15.94:0/336720668 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6120020070 con 0x7f6130070c50 2023-09-30T00:18:43.302 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.297+0000 7f6126ffd700 1 -- 172.21.15.94:0/336720668 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612001b6a0 con 0x7f6130070c50 2023-09-30T00:18:43.302 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.297+0000 7f6134948700 1 -- 172.21.15.94:0/336720668 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61301515d0 con 0x7f6130070c50 2023-09-30T00:18:43.302 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.297+0000 7f6134948700 1 -- 172.21.15.94:0/336720668 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61301441d0 con 0x7f6130070c50 2023-09-30T00:18:43.302 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.298+0000 7f6134948700 1 -- 172.21.15.94:0/336720668 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61300ae1e0 con 0x7f6130070c50 2023-09-30T00:18:43.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.298+0000 7f6126ffd700 1 -- 172.21.15.94:0/336720668 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6120022030 con 0x7f6130070c50 2023-09-30T00:18:43.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.298+0000 7f6126ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:43.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.299+0000 7f6126ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:43.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.299+0000 7f6126ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:43.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.299+0000 7f6126ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:43.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.299+0000 7f6126ffd700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6110098780 0x7f611009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:43.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.299+0000 7f6125d9b700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6110098780 0x7f611009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:43.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.299+0000 7f6126ffd700 1 -- 172.21.15.94:0/336720668 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f61200c1e50 con 0x7f6130070c50 2023-09-30T00:18:43.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.299+0000 7f6125d9b700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6110098780 0x7f611009ac30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6130144b60 tx=0x7f6118007ea0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:43.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.304+0000 7f6126ffd700 1 -- 172.21.15.94:0/336720668 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61300ae1e0 con 0x7f6130070c50 2023-09-30T00:18:43.526 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.525+0000 7f6134948700 1 -- 172.21.15.94:0/336720668 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f61300ae1e0 con 0x7f6130070c50 2023-09-30T00:18:43.527 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.525+0000 7f6126ffd700 1 -- 172.21.15.94:0/336720668 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6120098100 con 0x7f6130070c50 2023-09-30T00:18:43.527 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:43.527 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:43.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 10 mgrc shutdown 2023-09-30T00:18:43.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 1 -- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6110098780 msgr2=0x7f611009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:43.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6110098780 0x7f611009ac30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6130144b60 tx=0x7f6118007ea0 comp rx=0 tx=0).stop 2023-09-30T00:18:43.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 1 -- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6130070c50 msgr2=0x7f61300ad920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:43.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6130070c50 0x7f61300ad920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f61300a97e0 tx=0x7f6120016000 comp rx=0 tx=0).stop 2023-09-30T00:18:43.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 1 -- 172.21.15.94:0/336720668 shutdown_connections 2023-09-30T00:18:43.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6110098780 0x7f611009ac30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:43.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6130070c50 0x7f61300ad920 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:43.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f613013fd70 0x7f6130153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:43.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 1 --2- 172.21.15.94:0/336720668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61300b7160 0x7f61300ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:43.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 1 -- 172.21.15.94:0/336720668 >> 172.21.15.94:0/336720668 conn(0x7f613006a090 msgr2=0x7f613013d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:43.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 1 -- 172.21.15.94:0/336720668 shutdown_connections 2023-09-30T00:18:43.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:43.527+0000 7f6134948700 1 -- 172.21.15.94:0/336720668 wait complete. 2023-09-30T00:18:46.541 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:46.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.664+0000 7f005cbbc700 1 Processor -- start 2023-09-30T00:18:46.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.664+0000 7f005cbbc700 1 -- start start 2023-09-30T00:18:46.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.665+0000 7f005cbbc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f00580a94d0 0x7f00580a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:46.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.665+0000 7f005cbbc700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0058068200 con 0x7f00580a9ea0 2023-09-30T00:18:46.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.665+0000 7f005cbbc700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00580a24a0 con 0x7f0058067e30 2023-09-30T00:18:46.667 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.665+0000 7f005cbbc700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00580a22b0 con 0x7f00580a94d0 2023-09-30T00:18:46.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.665+0000 7f0056d9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f00580a94d0 0x7f00580a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:46.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.665+0000 7f0055d9b700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f00580a9ea0 0x7f0058149050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51758/0 (socket says 172.21.15.94:51758) 2023-09-30T00:18:46.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.665+0000 7f0055d9b700 1 -- 172.21.15.94:0/2778920846 learned_addr learned my addr 172.21.15.94:0/2778920846 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:46.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.665+0000 7f005559a700 1 -- 172.21.15.94:0/2778920846 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124527495 0 0) 0x7f0058068200 con 0x7f00580a9ea0 2023-09-30T00:18:46.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f005559a700 1 -- 172.21.15.94:0/2778920846 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0044003570 con 0x7f00580a9ea0 2023-09-30T00:18:46.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f0056d9d700 1 -- 172.21.15.94:0/2778920846 >> v1:172.21.15.121:6789/0 conn(0x7f0058067e30 legacy=0x7f005813fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:46.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f0056d9d700 1 -- 172.21.15.94:0/2778920846 >> v1:172.21.15.94:6789/0 conn(0x7f00580a9ea0 legacy=0x7f0058149050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:46.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f0056d9d700 1 -- 172.21.15.94:0/2778920846 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00580a2740 con 0x7f00580a94d0 2023-09-30T00:18:46.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f0056d9d700 1 --2- 172.21.15.94:0/2778920846 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f00580a94d0 0x7f00580a1d70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f00580a2930 tx=0x7f00500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b24c60951f82ee26 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:46.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f005559a700 1 -- 172.21.15.94:0/2778920846 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050011630 con 0x7f00580a94d0 2023-09-30T00:18:46.668 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f005559a700 1 -- 172.21.15.94:0/2778920846 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0050011790 con 0x7f00580a94d0 2023-09-30T00:18:46.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f005559a700 1 -- 172.21.15.94:0/2778920846 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050011b10 con 0x7f00580a94d0 2023-09-30T00:18:46.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f005cbbc700 1 -- 172.21.15.94:0/2778920846 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f00580a94d0 msgr2=0x7f00580a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:46.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f005cbbc700 1 --2- 172.21.15.94:0/2778920846 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f00580a94d0 0x7f00580a1d70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f00580a2930 tx=0x7f00500091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:46.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f005cbbc700 1 -- 172.21.15.94:0/2778920846 shutdown_connections 2023-09-30T00:18:46.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f005cbbc700 1 --2- 172.21.15.94:0/2778920846 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f00580a94d0 0x7f00580a1d70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:46.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.666+0000 7f005cbbc700 1 -- 172.21.15.94:0/2778920846 >> 172.21.15.94:0/2778920846 conn(0x7f00580702f0 msgr2=0x7f00580706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:46.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.667+0000 7f005cbbc700 1 -- 172.21.15.94:0/2778920846 shutdown_connections 2023-09-30T00:18:46.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.667+0000 7f005cbbc700 1 -- 172.21.15.94:0/2778920846 wait complete. 2023-09-30T00:18:46.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.667+0000 7f005cbbc700 1 Processor -- start 2023-09-30T00:18:46.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.667+0000 7f005cbbc700 1 -- start start 2023-09-30T00:18:46.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f005cbbc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0058067e30 0x7f00581440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:46.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f005cbbc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00580a94d0 0x7f0058141f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:46.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f005cbbc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0058144600 0x7f00581424a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:46.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f005cbbc700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00581429e0 con 0x7f0058067e30 2023-09-30T00:18:46.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f005cbbc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00580b47d0 con 0x7f00580a94d0 2023-09-30T00:18:46.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f005cbbc700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0058142bd0 con 0x7f0058144600 2023-09-30T00:18:46.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f005659c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0058067e30 0x7f00581440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:46.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f0056d9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0058144600 0x7f00581424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:46.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f005659c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0058067e30 0x7f00581440c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33492/0 (socket says 172.21.15.94:33492) 2023-09-30T00:18:46.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f005659c700 1 -- 172.21.15.94:0/3474306009 learned_addr learned my addr 172.21.15.94:0/3474306009 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:46.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f0055d9b700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00580a94d0 0x7f0058141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:46.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f0056d9d700 1 -- 172.21.15.94:0/3474306009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00580a94d0 msgr2=0x7f0058141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:46.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f0056d9d700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00580a94d0 0x7f0058141f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:46.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f0056d9d700 1 -- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0058067e30 msgr2=0x7f00581440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:46.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f0056d9d700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0058067e30 0x7f00581440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:46.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.668+0000 7f0056d9d700 1 -- 172.21.15.94:0/3474306009 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0050009420 con 0x7f0058144600 2023-09-30T00:18:46.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.669+0000 7f0056d9d700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0058144600 0x7f00581424a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f00580b4a50 tx=0x7f0050017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:46.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.669+0000 7f004e7fc700 1 -- 172.21.15.94:0/3474306009 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005001f070 con 0x7f0058144600 2023-09-30T00:18:46.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.669+0000 7f004e7fc700 1 -- 172.21.15.94:0/3474306009 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0050011900 con 0x7f0058144600 2023-09-30T00:18:46.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.669+0000 7f005cbbc700 1 -- 172.21.15.94:0/3474306009 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0058141ac0 con 0x7f0058144600 2023-09-30T00:18:46.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.669+0000 7f004e7fc700 1 -- 172.21.15.94:0/3474306009 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050025690 con 0x7f0058144600 2023-09-30T00:18:46.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.669+0000 7f005cbbc700 1 -- 172.21.15.94:0/3474306009 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00580a5ad0 con 0x7f0058144600 2023-09-30T00:18:46.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.670+0000 7f005cbbc700 1 -- 172.21.15.94:0/3474306009 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005806fbd0 con 0x7f0058144600 2023-09-30T00:18:46.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.671+0000 7f004e7fc700 1 -- 172.21.15.94:0/3474306009 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f0050023070 con 0x7f0058144600 2023-09-30T00:18:46.674 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.671+0000 7f004e7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.671+0000 7f004e7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.671+0000 7f004e7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.671+0000 7f004e7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.671+0000 7f004e7fc700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f00380987b0 0x7f003809ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.671+0000 7f005659c700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f00380987b0 0x7f003809ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.671+0000 7f004e7fc700 1 -- 172.21.15.94:0/3474306009 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f00500c1950 con 0x7f0058144600 2023-09-30T00:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.672+0000 7f005659c700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f00380987b0 0x7f003809ac60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0048007fd0 tx=0x7f0048002760 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:46.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.676+0000 7f004e7fc700 1 -- 172.21.15.94:0/3474306009 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f005008f020 con 0x7f0058144600 2023-09-30T00:18:46.898 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.897+0000 7f005cbbc700 1 -- 172.21.15.94:0/3474306009 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f00580b5f00 con 0x7f0058144600 2023-09-30T00:18:46.898 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.897+0000 7f004e7fc700 1 -- 172.21.15.94:0/3474306009 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0050092640 con 0x7f0058144600 2023-09-30T00:18:46.898 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:46.898 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:46.900 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.899+0000 7f005cbbc700 10 mgrc shutdown 2023-09-30T00:18:46.900 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.899+0000 7f005cbbc700 1 -- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f00380987b0 msgr2=0x7f003809ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:46.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.900+0000 7f005cbbc700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f00380987b0 0x7f003809ac60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0048007fd0 tx=0x7f0048002760 comp rx=0 tx=0).stop 2023-09-30T00:18:46.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.900+0000 7f005cbbc700 1 -- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0058144600 msgr2=0x7f00581424a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:46.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.900+0000 7f005cbbc700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0058144600 0x7f00581424a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f00580b4a50 tx=0x7f0050017000 comp rx=0 tx=0).stop 2023-09-30T00:18:46.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.900+0000 7f005cbbc700 1 -- 172.21.15.94:0/3474306009 shutdown_connections 2023-09-30T00:18:46.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.900+0000 7f005cbbc700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f00380987b0 0x7f003809ac60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:46.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.900+0000 7f005cbbc700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0058144600 0x7f00581424a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:46.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.900+0000 7f005cbbc700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0058067e30 0x7f00581440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:46.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.900+0000 7f005cbbc700 1 --2- 172.21.15.94:0/3474306009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00580a94d0 0x7f0058141f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:46.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.900+0000 7f005cbbc700 1 -- 172.21.15.94:0/3474306009 >> 172.21.15.94:0/3474306009 conn(0x7f00580702f0 msgr2=0x7f00581402a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:46.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.900+0000 7f005cbbc700 1 -- 172.21.15.94:0/3474306009 shutdown_connections 2023-09-30T00:18:46.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:46.900+0000 7f005cbbc700 1 -- 172.21.15.94:0/3474306009 wait complete. 2023-09-30T00:18:49.913 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:50.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.037+0000 7fc4fc495700 1 Processor -- start 2023-09-30T00:18:50.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.037+0000 7fc4fc495700 1 -- start start 2023-09-30T00:18:50.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4fc495700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc4f40b7160 0x7fc4f40a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:50.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4fc495700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f40a1310 con 0x7fc4f413fd70 2023-09-30T00:18:50.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4fc495700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f4071020 con 0x7fc4f4070c50 2023-09-30T00:18:50.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4fc495700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f40b76a0 con 0x7fc4f40b7160 2023-09-30T00:18:50.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4faa32700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc4f413fd70 0x7fc4f414b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51762/0 (socket says 172.21.15.94:51762) 2023-09-30T00:18:50.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4faa32700 1 -- 172.21.15.94:0/1344499685 learned_addr learned my addr 172.21.15.94:0/1344499685 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:50.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f9a30700 1 --2- 172.21.15.94:0/1344499685 >> v2:172.21.15.121:3300/0 conn(0x7fc4f40b7160 0x7fc4f40a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:50.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827478091 0 0) 0x7fc4f0001110 con 0x7fc4f413fd70 2023-09-30T00:18:50.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4e0003570 con 0x7fc4f413fd70 2023-09-30T00:18:50.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045118099 0 0) 0x7fc4f4071020 con 0x7fc4f4070c50 2023-09-30T00:18:50.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4f0001110 con 0x7fc4f4070c50 2023-09-30T00:18:50.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1842873782 0 0) 0x7fc4e0003570 con 0x7fc4f413fd70 2023-09-30T00:18:50.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 >> v2:172.21.15.121:3300/0 conn(0x7fc4f40b7160 msgr2=0x7fc4f40a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:50.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 --2- 172.21.15.94:0/1344499685 >> v2:172.21.15.121:3300/0 conn(0x7fc4f40b7160 0x7fc4f40a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 >> v1:172.21.15.121:6789/0 conn(0x7fc4f4070c50 legacy=0x7fc4f40a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:50.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f40b7940 con 0x7fc4f413fd70 2023-09-30T00:18:50.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3153706020 0 0) 0x7fc4f0001600 con 0x7fc4f413fd70 2023-09-30T00:18:50.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 >> v1:172.21.15.94:6789/0 conn(0x7fc4f413fd70 legacy=0x7fc4f414b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:50.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 --2- 172.21.15.94:0/1344499685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4e0003bf0 0x7fc4e0003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:50.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 --2- 172.21.15.94:0/1344499685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4e0004810 0x7fc4e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:50.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 --2- 172.21.15.94:0/1344499685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4e00051c0 0x7fc4e000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:50.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40b76a0 con 0x7fc4e00051c0 2023-09-30T00:18:50.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e000c0b0 con 0x7fc4e0004810 2023-09-30T00:18:50.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.038+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e000c270 con 0x7fc4e0003bf0 2023-09-30T00:18:50.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4fa231700 1 --2- 172.21.15.94:0/1344499685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4e0003bf0 0x7fc4e0003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:50.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4faa32700 1 --2- 172.21.15.94:0/1344499685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4e00051c0 0x7fc4e000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:50.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4f9a30700 1 --2- 172.21.15.94:0/1344499685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4e0004810 0x7fc4e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4faa32700 1 -- 172.21.15.94:0/1344499685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4e0003bf0 msgr2=0x7fc4e0003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4faa32700 1 --2- 172.21.15.94:0/1344499685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4e0003bf0 0x7fc4e0003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4faa32700 1 -- 172.21.15.94:0/1344499685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4e0004810 msgr2=0x7fc4e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4faa32700 1 --2- 172.21.15.94:0/1344499685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4e0004810 0x7fc4e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4faa32700 1 -- 172.21.15.94:0/1344499685 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e0003570 con 0x7fc4e00051c0 2023-09-30T00:18:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4fa231700 1 --2- 172.21.15.94:0/1344499685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4e0003bf0 0x7fc4e0003fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4faa32700 1 -- 172.21.15.94:0/1344499685 reap_dead start 2023-09-30T00:18:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4faa32700 1 --2- 172.21.15.94:0/1344499685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4e00051c0 0x7fc4e000ba60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc4f000c5f0 tx=0x7fc4f40b64d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02ed55cc4bab3bc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f000cdf0 con 0x7fc4e00051c0 2023-09-30T00:18:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f0005920 con 0x7fc4e00051c0 2023-09-30T00:18:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.039+0000 7fc4f922f700 1 -- 172.21.15.94:0/1344499685 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0005cc0 con 0x7fc4e00051c0 2023-09-30T00:18:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.040+0000 7fc4fc495700 1 -- 172.21.15.94:0/1344499685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4e00051c0 msgr2=0x7fc4e000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.040+0000 7fc4fc495700 1 --2- 172.21.15.94:0/1344499685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4e00051c0 0x7fc4e000ba60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc4f000c5f0 tx=0x7fc4f40b64d0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.040+0000 7fc4fc495700 1 -- 172.21.15.94:0/1344499685 shutdown_connections 2023-09-30T00:18:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.040+0000 7fc4fc495700 1 --2- 172.21.15.94:0/1344499685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4e00051c0 0x7fc4e000ba60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.040+0000 7fc4fc495700 1 -- 172.21.15.94:0/1344499685 >> 172.21.15.94:0/1344499685 conn(0x7fc4f406a090 msgr2=0x7fc4f40702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.040+0000 7fc4fc495700 1 -- 172.21.15.94:0/1344499685 shutdown_connections 2023-09-30T00:18:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.040+0000 7fc4fc495700 1 -- 172.21.15.94:0/1344499685 wait complete. 2023-09-30T00:18:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.041+0000 7fc4fc495700 1 Processor -- start 2023-09-30T00:18:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.041+0000 7fc4fc495700 1 -- start start 2023-09-30T00:18:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.041+0000 7fc4fc495700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4e00051c0 0x7fc4f4153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.041+0000 7fc4fc495700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4f40a8920 0x7fc4f413a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.041+0000 7fc4fc495700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4f40a8cf0 0x7fc4f413aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.041+0000 7fc4fc495700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40a5e20 con 0x7fc4f40a8cf0 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.041+0000 7fc4fc495700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f4154160 con 0x7fc4e00051c0 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.041+0000 7fc4fc495700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f40a6bd0 con 0x7fc4f40a8920 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.041+0000 7fc4f9a30700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4f40a8920 0x7fc4f413a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.041+0000 7fc4faa32700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4f40a8cf0 0x7fc4f413aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.041+0000 7fc4f9a30700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4f40a8920 0x7fc4f413a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:47386/0 (socket says 172.21.15.94:47386) 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.041+0000 7fc4f9a30700 1 -- 172.21.15.94:0/304757457 learned_addr learned my addr 172.21.15.94:0/304757457 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.042+0000 7fc4fa231700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4e00051c0 0x7fc4f4153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.042+0000 7fc4f9a30700 1 -- 172.21.15.94:0/304757457 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4e00051c0 msgr2=0x7fc4f4153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.042+0000 7fc4f9a30700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4e00051c0 0x7fc4f4153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.042+0000 7fc4f9a30700 1 -- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4f40a8cf0 msgr2=0x7fc4f413aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.042+0000 7fc4f9a30700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4f40a8cf0 0x7fc4f413aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.042+0000 7fc4f9a30700 1 -- 172.21.15.94:0/304757457 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f41543c0 con 0x7fc4f40a8920 2023-09-30T00:18:50.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.042+0000 7fc4f9a30700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4f40a8920 0x7fc4f413a570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4006ac0 tx=0x7fc4e40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:50.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.042+0000 7fc4ea7fc700 1 -- 172.21.15.94:0/304757457 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e400f940 con 0x7fc4f40a8920 2023-09-30T00:18:50.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.042+0000 7fc4ea7fc700 1 -- 172.21.15.94:0/304757457 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e400faa0 con 0x7fc4f40a8920 2023-09-30T00:18:50.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.042+0000 7fc4fc495700 1 -- 172.21.15.94:0/304757457 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f4154660 con 0x7fc4f40a8920 2023-09-30T00:18:50.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.042+0000 7fc4ea7fc700 1 -- 172.21.15.94:0/304757457 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e400fe20 con 0x7fc4f40a8920 2023-09-30T00:18:50.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.042+0000 7fc4fc495700 1 -- 172.21.15.94:0/304757457 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f40b5ba0 con 0x7fc4f40a8920 2023-09-30T00:18:50.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.043+0000 7fc4fc495700 1 -- 172.21.15.94:0/304757457 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f4071630 con 0x7fc4f40a8920 2023-09-30T00:18:50.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.044+0000 7fc4ea7fc700 1 -- 172.21.15.94:0/304757457 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc4e4014650 con 0x7fc4f40a8920 2023-09-30T00:18:50.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.044+0000 7fc4ea7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:50.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.045+0000 7fc4ea7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:50.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.045+0000 7fc4ea7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:50.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.045+0000 7fc4ea7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.045+0000 7fc4ea7fc700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc4d8098780 0x7fc4d809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.045+0000 7fc4fa231700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc4d8098780 0x7fc4d809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.045+0000 7fc4ea7fc700 1 -- 172.21.15.94:0/304757457 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fc4e4018030 con 0x7fc4f40a8920 2023-09-30T00:18:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.045+0000 7fc4fa231700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc4d8098780 0x7fc4d809ac30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0018f0 tx=0x7fc4ec0017e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.049+0000 7fc4ea7fc700 1 -- 172.21.15.94:0/304757457 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc4e40c5070 con 0x7fc4f40a8920 2023-09-30T00:18:50.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.272+0000 7fc4fc495700 1 -- 172.21.15.94:0/304757457 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc4f40a2070 con 0x7fc4f40a8920 2023-09-30T00:18:50.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.272+0000 7fc4ea7fc700 1 -- 172.21.15.94:0/304757457 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc4e4022020 con 0x7fc4f40a8920 2023-09-30T00:18:50.273 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:50.273 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:50.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 10 mgrc shutdown 2023-09-30T00:18:50.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 1 -- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc4d8098780 msgr2=0x7fc4d809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:50.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc4d8098780 0x7fc4d809ac30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0018f0 tx=0x7fc4ec0017e0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 1 -- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4f40a8920 msgr2=0x7fc4f413a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:50.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4f40a8920 0x7fc4f413a570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4006ac0 tx=0x7fc4e40091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 1 -- 172.21.15.94:0/304757457 shutdown_connections 2023-09-30T00:18:50.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc4d8098780 0x7fc4d809ac30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4f40a8cf0 0x7fc4f413aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4e00051c0 0x7fc4f4153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 1 --2- 172.21.15.94:0/304757457 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4f40a8920 0x7fc4f413a570 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:50.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 1 -- 172.21.15.94:0/304757457 >> 172.21.15.94:0/304757457 conn(0x7fc4f406a090 msgr2=0x7fc4f4141ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:50.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 1 -- 172.21.15.94:0/304757457 shutdown_connections 2023-09-30T00:18:50.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:50.274+0000 7fc4fc495700 1 -- 172.21.15.94:0/304757457 wait complete. 2023-09-30T00:18:53.287 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:53.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.412+0000 7ff279823700 1 Processor -- start 2023-09-30T00:18:53.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.412+0000 7ff279823700 1 -- start start 2023-09-30T00:18:53.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.413+0000 7ff279823700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff274070c50 0x7ff2740a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:53.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.413+0000 7ff279823700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2740a1310 con 0x7ff2740b7160 2023-09-30T00:18:53.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.413+0000 7ff279823700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff274071020 con 0x7ff27413fd70 2023-09-30T00:18:53.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.413+0000 7ff279823700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2740b76a0 con 0x7ff274070c50 2023-09-30T00:18:53.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.413+0000 7ff272ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff274070c50 0x7ff2740a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:53.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.413+0000 7ff2727fc700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff2740b7160 0x7ff2740a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51776/0 (socket says 172.21.15.94:51776) 2023-09-30T00:18:53.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.413+0000 7ff2727fc700 1 -- 172.21.15.94:0/3047081466 learned_addr learned my addr 172.21.15.94:0/3047081466 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:18:53.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.413+0000 7ff272ffd700 1 -- 172.21.15.94:0/3047081466 >> v1:172.21.15.121:6789/0 conn(0x7ff27413fd70 legacy=0x7ff27414b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:53.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.413+0000 7ff272ffd700 1 -- 172.21.15.94:0/3047081466 >> v1:172.21.15.94:6789/0 conn(0x7ff2740b7160 legacy=0x7ff2740a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:53.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.413+0000 7ff271ffb700 1 -- 172.21.15.94:0/3047081466 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615492277 0 0) 0x7ff268001110 con 0x7ff2740b7160 2023-09-30T00:18:53.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.413+0000 7ff272ffd700 1 -- 172.21.15.94:0/3047081466 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2740b7940 con 0x7ff274070c50 2023-09-30T00:18:53.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.414+0000 7ff272ffd700 1 --2- 172.21.15.94:0/3047081466 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff274070c50 0x7ff2740a0dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff264009e90 tx=0x7ff264009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=239a5213c29c5bf4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:53.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.414+0000 7ff271ffb700 1 -- 172.21.15.94:0/3047081466 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26400e070 con 0x7ff274070c50 2023-09-30T00:18:53.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.414+0000 7ff271ffb700 1 -- 172.21.15.94:0/3047081466 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26400bc90 con 0x7ff274070c50 2023-09-30T00:18:53.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.414+0000 7ff271ffb700 1 -- 172.21.15.94:0/3047081466 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff264012680 con 0x7ff274070c50 2023-09-30T00:18:53.418 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.414+0000 7ff279823700 1 -- 172.21.15.94:0/3047081466 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff274070c50 msgr2=0x7ff2740a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:53.418 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.414+0000 7ff279823700 1 --2- 172.21.15.94:0/3047081466 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff274070c50 0x7ff2740a0dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff264009e90 tx=0x7ff264009780 comp rx=0 tx=0).stop 2023-09-30T00:18:53.418 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.414+0000 7ff279823700 1 -- 172.21.15.94:0/3047081466 shutdown_connections 2023-09-30T00:18:53.418 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.414+0000 7ff279823700 1 --2- 172.21.15.94:0/3047081466 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff274070c50 0x7ff2740a0dd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:53.418 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.414+0000 7ff279823700 1 -- 172.21.15.94:0/3047081466 >> 172.21.15.94:0/3047081466 conn(0x7ff27406a090 msgr2=0x7ff2740702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:53.418 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.414+0000 7ff279823700 1 -- 172.21.15.94:0/3047081466 shutdown_connections 2023-09-30T00:18:53.418 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.414+0000 7ff279823700 1 -- 172.21.15.94:0/3047081466 wait complete. 2023-09-30T00:18:53.418 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.415+0000 7ff279823700 1 Processor -- start 2023-09-30T00:18:53.418 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.415+0000 7ff279823700 1 -- start start 2023-09-30T00:18:53.419 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff279823700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff274070c50 0x7ff274146cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:53.419 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff279823700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff2740b7160 0x7ff274140a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:53.419 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff279823700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff27413fd70 0x7ff274147df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:53.419 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff279823700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27413cc20 con 0x7ff274070c50 2023-09-30T00:18:53.419 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff279823700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2741468c0 con 0x7ff27413fd70 2023-09-30T00:18:53.419 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff279823700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2741484f0 con 0x7ff2740b7160 2023-09-30T00:18:53.419 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff272ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff274070c50 0x7ff274146cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:53.419 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff2727fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff2740b7160 0x7ff274140a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff272ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff274070c50 0x7ff274146cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33518/0 (socket says 172.21.15.94:33518) 2023-09-30T00:18:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff272ffd700 1 -- 172.21.15.94:0/1202130513 learned_addr learned my addr 172.21.15.94:0/1202130513 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff2737fe700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff27413fd70 0x7ff274147df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff272ffd700 1 -- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff2740b7160 msgr2=0x7ff274140a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff272ffd700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff2740b7160 0x7ff274140a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff272ffd700 1 -- 172.21.15.94:0/1202130513 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff27413fd70 msgr2=0x7ff274147df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff272ffd700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff27413fd70 0x7ff274147df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff272ffd700 1 -- 172.21.15.94:0/1202130513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff264009a20 con 0x7ff274070c50 2023-09-30T00:18:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff2737fe700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff27413fd70 0x7ff274147df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:18:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.416+0000 7ff272ffd700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff274070c50 0x7ff274146cf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff2740a97e0 tx=0x7ff264019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.417+0000 7ff25affd700 1 -- 172.21.15.94:0/1202130513 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26401b420 con 0x7ff274070c50 2023-09-30T00:18:53.420 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.417+0000 7ff25affd700 1 -- 172.21.15.94:0/1202130513 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26401b5a0 con 0x7ff274070c50 2023-09-30T00:18:53.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.417+0000 7ff279823700 1 -- 172.21.15.94:0/1202130513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2741487b0 con 0x7ff274070c50 2023-09-30T00:18:53.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.417+0000 7ff25affd700 1 -- 172.21.15.94:0/1202130513 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26401b960 con 0x7ff274070c50 2023-09-30T00:18:53.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.417+0000 7ff279823700 1 -- 172.21.15.94:0/1202130513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2740a62a0 con 0x7ff274070c50 2023-09-30T00:18:53.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.418+0000 7ff279823700 1 -- 172.21.15.94:0/1202130513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff274146a50 con 0x7ff274070c50 2023-09-30T00:18:53.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.419+0000 7ff25affd700 1 -- 172.21.15.94:0/1202130513 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ff264003020 con 0x7ff274070c50 2023-09-30T00:18:53.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.419+0000 7ff25affd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:53.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.420+0000 7ff25affd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:53.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.420+0000 7ff25affd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:53.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.420+0000 7ff25affd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:53.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.420+0000 7ff25affd700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff2600987b0 0x7ff26009ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:53.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.420+0000 7ff2727fc700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff2600987b0 0x7ff26009ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:53.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.420+0000 7ff25affd700 1 -- 172.21.15.94:0/1202130513 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7ff2640c22e0 con 0x7ff274070c50 2023-09-30T00:18:53.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.420+0000 7ff2727fc700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff2600987b0 0x7ff26009ac60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff27413ec00 tx=0x7ff268007b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:53.423 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.423+0000 7ff25affd700 1 -- 172.21.15.94:0/1202130513 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff26408f910 con 0x7ff274070c50 2023-09-30T00:18:53.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.644+0000 7ff279823700 1 -- 172.21.15.94:0/1202130513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff274146a00 con 0x7ff274070c50 2023-09-30T00:18:53.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.644+0000 7ff25affd700 1 -- 172.21.15.94:0/1202130513 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff264092f30 con 0x7ff274070c50 2023-09-30T00:18:53.646 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:53.646 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:53.648 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.646+0000 7ff279823700 10 mgrc shutdown 2023-09-30T00:18:53.648 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.646+0000 7ff279823700 1 -- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff2600987b0 msgr2=0x7ff26009ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:53.648 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.646+0000 7ff279823700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff2600987b0 0x7ff26009ac60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff27413ec00 tx=0x7ff268007b70 comp rx=0 tx=0).stop 2023-09-30T00:18:53.648 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.646+0000 7ff279823700 1 -- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff274070c50 msgr2=0x7ff274146cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:53.648 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.646+0000 7ff279823700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff274070c50 0x7ff274146cf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff2740a97e0 tx=0x7ff264019000 comp rx=0 tx=0).stop 2023-09-30T00:18:53.648 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.646+0000 7ff279823700 1 -- 172.21.15.94:0/1202130513 shutdown_connections 2023-09-30T00:18:53.648 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.646+0000 7ff279823700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff2600987b0 0x7ff26009ac60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:53.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.646+0000 7ff279823700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff27413fd70 0x7ff274147df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:53.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.646+0000 7ff279823700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff274070c50 0x7ff274146cf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:53.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.646+0000 7ff279823700 1 --2- 172.21.15.94:0/1202130513 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff2740b7160 0x7ff274140a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:53.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.646+0000 7ff279823700 1 -- 172.21.15.94:0/1202130513 >> 172.21.15.94:0/1202130513 conn(0x7ff27406a090 msgr2=0x7ff27413d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:53.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.646+0000 7ff279823700 1 -- 172.21.15.94:0/1202130513 shutdown_connections 2023-09-30T00:18:53.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:53.647+0000 7ff279823700 1 -- 172.21.15.94:0/1202130513 wait complete. 2023-09-30T00:18:56.659 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:18:56.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.784+0000 7fcba04fe700 1 Processor -- start 2023-09-30T00:18:56.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.784+0000 7fcba04fe700 1 -- start start 2023-09-30T00:18:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.785+0000 7fcba04fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 0x7fcb980a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.785+0000 7fcba04fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcb980b7160 0x7fcb980a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.785+0000 7fcba04fe700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb980a1310 con 0x7fcb9813fd70 2023-09-30T00:18:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.785+0000 7fcba04fe700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb98071020 con 0x7fcb980b7160 2023-09-30T00:18:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.785+0000 7fcba04fe700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb980b76a0 con 0x7fcb98070c50 2023-09-30T00:18:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.785+0000 7fcb9e29a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 0x7fcb980a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.785+0000 7fcb9e29a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 0x7fcb980a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:57948/0 (socket says 172.21.15.94:57948) 2023-09-30T00:18:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.785+0000 7fcb9e29a700 1 -- 172.21.15.94:0/2091930695 learned_addr learned my addr 172.21.15.94:0/2091930695 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.785+0000 7fcb9da99700 1 --2- 172.21.15.94:0/2091930695 >> v2:172.21.15.121:3300/0 conn(0x7fcb980b7160 0x7fcb980a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.785+0000 7fcb9e29a700 1 -- 172.21.15.94:0/2091930695 >> v2:172.21.15.121:3300/0 conn(0x7fcb980b7160 msgr2=0x7fcb980a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:18:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.785+0000 7fcb9e29a700 1 --2- 172.21.15.94:0/2091930695 >> v2:172.21.15.121:3300/0 conn(0x7fcb980b7160 0x7fcb980a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.785+0000 7fcb9e29a700 1 -- 172.21.15.94:0/2091930695 >> v1:172.21.15.121:6789/0 conn(0x7fcb9813fd70 legacy=0x7fcb980b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.786+0000 7fcb9e29a700 1 -- 172.21.15.94:0/2091930695 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb980b7940 con 0x7fcb98070c50 2023-09-30T00:18:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.786+0000 7fcb9e29a700 1 --2- 172.21.15.94:0/2091930695 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 0x7fcb980a0dd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcb940099b0 tx=0x7fcb94009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdedb807abe66c63 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.786+0000 7fcb9d298700 1 -- 172.21.15.94:0/2091930695 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94009ea0 con 0x7fcb98070c50 2023-09-30T00:18:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.786+0000 7fcb9d298700 1 -- 172.21.15.94:0/2091930695 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9400b8f0 con 0x7fcb98070c50 2023-09-30T00:18:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.786+0000 7fcb9d298700 1 -- 172.21.15.94:0/2091930695 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9400bc90 con 0x7fcb98070c50 2023-09-30T00:18:56.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.786+0000 7fcba04fe700 1 -- 172.21.15.94:0/2091930695 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 msgr2=0x7fcb980a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:56.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.786+0000 7fcba04fe700 1 --2- 172.21.15.94:0/2091930695 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 0x7fcb980a0dd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcb940099b0 tx=0x7fcb94009250 comp rx=0 tx=0).stop 2023-09-30T00:18:56.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.786+0000 7fcba04fe700 1 -- 172.21.15.94:0/2091930695 shutdown_connections 2023-09-30T00:18:56.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.786+0000 7fcba04fe700 1 --2- 172.21.15.94:0/2091930695 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 0x7fcb980a0dd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:56.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.786+0000 7fcba04fe700 1 --2- 172.21.15.94:0/2091930695 >> v2:172.21.15.121:3300/0 conn(0x7fcb980b7160 0x7fcb980a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:56.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.786+0000 7fcba04fe700 1 -- 172.21.15.94:0/2091930695 >> 172.21.15.94:0/2091930695 conn(0x7fcb9806a090 msgr2=0x7fcb980702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:56.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.786+0000 7fcba04fe700 1 -- 172.21.15.94:0/2091930695 shutdown_connections 2023-09-30T00:18:56.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.787+0000 7fcba04fe700 1 -- 172.21.15.94:0/2091930695 wait complete. 2023-09-30T00:18:56.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.787+0000 7fcba04fe700 1 Processor -- start 2023-09-30T00:18:56.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.787+0000 7fcba04fe700 1 -- start start 2023-09-30T00:18:56.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcba04fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 0x7fcb980a8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:56.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcba04fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb980b7160 0x7fcb98153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:56.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcba04fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9813fd70 0x7fcb9813a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:56.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcba04fe700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb980a6810 con 0x7fcb9813fd70 2023-09-30T00:18:56.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcba04fe700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb980a5b40 con 0x7fcb980b7160 2023-09-30T00:18:56.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcba04fe700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb98154590 con 0x7fcb98070c50 2023-09-30T00:18:56.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcb9e29a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 0x7fcb980a8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:56.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcb9e29a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 0x7fcb980a8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:57954/0 (socket says 172.21.15.94:57954) 2023-09-30T00:18:56.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcb9e29a700 1 -- 172.21.15.94:0/1107924460 learned_addr learned my addr 172.21.15.94:0/1107924460 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:18:56.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcb9ea9b700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9813fd70 0x7fcb9813a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:56.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcb9da99700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb980b7160 0x7fcb98153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:56.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcb9e29a700 1 -- 172.21.15.94:0/1107924460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb980b7160 msgr2=0x7fcb98153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:56.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcb9e29a700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb980b7160 0x7fcb98153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:56.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcb9e29a700 1 -- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9813fd70 msgr2=0x7fcb9813a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:56.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcb9e29a700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9813fd70 0x7fcb9813a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:56.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcb9e29a700 1 -- 172.21.15.94:0/1107924460 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb940094f0 con 0x7fcb98070c50 2023-09-30T00:18:56.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcb9ea9b700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9813fd70 0x7fcb9813a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:18:56.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.788+0000 7fcb9e29a700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 0x7fcb980a8750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcb980a97e0 tx=0x7fcb94017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:56.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.789+0000 7fcb8e7fc700 1 -- 172.21.15.94:0/1107924460 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9400ba60 con 0x7fcb98070c50 2023-09-30T00:18:56.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.789+0000 7fcb8e7fc700 1 -- 172.21.15.94:0/1107924460 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9400bbe0 con 0x7fcb98070c50 2023-09-30T00:18:56.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.789+0000 7fcba04fe700 1 -- 172.21.15.94:0/1107924460 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb98154850 con 0x7fcb98070c50 2023-09-30T00:18:56.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.789+0000 7fcb8e7fc700 1 -- 172.21.15.94:0/1107924460 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94026680 con 0x7fcb98070c50 2023-09-30T00:18:56.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.789+0000 7fcba04fe700 1 -- 172.21.15.94:0/1107924460 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb98143ad0 con 0x7fcb98070c50 2023-09-30T00:18:56.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.790+0000 7fcba04fe700 1 -- 172.21.15.94:0/1107924460 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb98071630 con 0x7fcb98070c50 2023-09-30T00:18:56.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.791+0000 7fcb8e7fc700 1 -- 172.21.15.94:0/1107924460 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fcb94023030 con 0x7fcb98070c50 2023-09-30T00:18:56.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.791+0000 7fcb8e7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:18:56.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.791+0000 7fcb8e7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:18:56.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.791+0000 7fcb8e7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:56.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.791+0000 7fcb8e7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:18:56.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.791+0000 7fcb8e7fc700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcb7c098780 0x7fcb7c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:18:56.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.791+0000 7fcb9da99700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcb7c098780 0x7fcb7c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:18:56.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.791+0000 7fcb8e7fc700 1 -- 172.21.15.94:0/1107924460 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fcb940c1c70 con 0x7fcb98070c50 2023-09-30T00:18:56.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.792+0000 7fcb9da99700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcb7c098780 0x7fcb7c09ac30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcb980a9100 tx=0x7fcb880091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:18:56.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:56.796+0000 7fcb8e7fc700 1 -- 172.21.15.94:0/1107924460 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcb9408f340 con 0x7fcb98070c50 2023-09-30T00:18:57.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.020+0000 7fcba04fe700 1 -- 172.21.15.94:0/1107924460 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcb980a2070 con 0x7fcb98070c50 2023-09-30T00:18:57.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.020+0000 7fcb8e7fc700 1 -- 172.21.15.94:0/1107924460 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcb9402e3a0 con 0x7fcb98070c50 2023-09-30T00:18:57.021 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:18:57.021 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:18:57.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.022+0000 7fcba04fe700 10 mgrc shutdown 2023-09-30T00:18:57.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.023+0000 7fcba04fe700 1 -- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcb7c098780 msgr2=0x7fcb7c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:57.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.023+0000 7fcba04fe700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcb7c098780 0x7fcb7c09ac30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcb980a9100 tx=0x7fcb880091a0 comp rx=0 tx=0).stop 2023-09-30T00:18:57.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.023+0000 7fcba04fe700 1 -- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 msgr2=0x7fcb980a8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:18:57.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.023+0000 7fcba04fe700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 0x7fcb980a8750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcb980a97e0 tx=0x7fcb94017000 comp rx=0 tx=0).stop 2023-09-30T00:18:57.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.023+0000 7fcba04fe700 1 -- 172.21.15.94:0/1107924460 shutdown_connections 2023-09-30T00:18:57.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.023+0000 7fcba04fe700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcb7c098780 0x7fcb7c09ac30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:57.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.023+0000 7fcba04fe700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb98070c50 0x7fcb980a8750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:57.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.023+0000 7fcba04fe700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9813fd70 0x7fcb9813a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:57.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.023+0000 7fcba04fe700 1 --2- 172.21.15.94:0/1107924460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb980b7160 0x7fcb98153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:18:57.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.023+0000 7fcba04fe700 1 -- 172.21.15.94:0/1107924460 >> 172.21.15.94:0/1107924460 conn(0x7fcb9806a090 msgr2=0x7fcb98146b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:18:57.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.023+0000 7fcba04fe700 1 -- 172.21.15.94:0/1107924460 shutdown_connections 2023-09-30T00:18:57.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:18:57.023+0000 7fcba04fe700 1 -- 172.21.15.94:0/1107924460 wait complete. 2023-09-30T00:19:00.036 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:00.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.160+0000 7f35ebdb1700 1 Processor -- start 2023-09-30T00:19:00.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.161+0000 7f35ebdb1700 1 -- start start 2023-09-30T00:19:00.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.161+0000 7f35ebdb1700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f35e406aaa0 0x7f35e40b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:00.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.161+0000 7f35ebdb1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e406ae70 con 0x7f35e40a3ff0 2023-09-30T00:19:00.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.161+0000 7f35ebdb1700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e40b40e0 con 0x7f35e409ca70 2023-09-30T00:19:00.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.161+0000 7f35ebdb1700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e40b48c0 con 0x7f35e406aaa0 2023-09-30T00:19:00.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.161+0000 7f35e934c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f35e40a3ff0 0x7f35e40b24b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49726/0 (socket says 172.21.15.94:49726) 2023-09-30T00:19:00.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.161+0000 7f35e934c700 1 -- 172.21.15.94:0/1249322074 learned_addr learned my addr 172.21.15.94:0/1249322074 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:19:00.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.161+0000 7f35e9b4d700 1 --2- 172.21.15.94:0/1249322074 >> v2:172.21.15.121:3300/0 conn(0x7f35e406aaa0 0x7f35e40b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:00.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013159120 0 0) 0x7f35e406ae70 con 0x7f35e40a3ff0 2023-09-30T00:19:00.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d0003570 con 0x7f35e40a3ff0 2023-09-30T00:19:00.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 905504393 0 0) 0x7f35d0003570 con 0x7f35e40a3ff0 2023-09-30T00:19:00.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 >> v2:172.21.15.121:3300/0 conn(0x7f35e406aaa0 msgr2=0x7f35e40b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:00.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 --2- 172.21.15.94:0/1249322074 >> v2:172.21.15.121:3300/0 conn(0x7f35e406aaa0 0x7f35e40b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:00.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 >> v1:172.21.15.121:6789/0 conn(0x7f35e409ca70 legacy=0x7f35e4147be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:00.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e40a45c0 con 0x7f35e40a3ff0 2023-09-30T00:19:00.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3153706020 0 0) 0x7f35d4001470 con 0x7f35e40a3ff0 2023-09-30T00:19:00.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e9b4d700 1 --2- 172.21.15.94:0/1249322074 >> v2:172.21.15.121:3300/0 conn(0x7f35e406aaa0 0x7f35e40b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:00.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 >> v1:172.21.15.94:6789/0 conn(0x7f35e40a3ff0 legacy=0x7f35e40b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:00.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 --2- 172.21.15.94:0/1249322074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35d0003a60 0x7f35d0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:00.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 --2- 172.21.15.94:0/1249322074 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35d0004810 0x7f35d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:00.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 --2- 172.21.15.94:0/1249322074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d00051c0 0x7f35d000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:00.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e40b48c0 con 0x7f35d00051c0 2023-09-30T00:19:00.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d000c0d0 con 0x7f35d0003a60 2023-09-30T00:19:00.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d000c290 con 0x7f35d0004810 2023-09-30T00:19:00.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e934c700 1 --2- 172.21.15.94:0/1249322074 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35d0004810 0x7f35d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:00.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35ea34e700 1 --2- 172.21.15.94:0/1249322074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d00051c0 0x7f35d000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:00.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e9b4d700 1 --2- 172.21.15.94:0/1249322074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35d0003a60 0x7f35d0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:00.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e934c700 1 -- 172.21.15.94:0/1249322074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35d0003a60 msgr2=0x7f35d0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:00.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e934c700 1 --2- 172.21.15.94:0/1249322074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35d0003a60 0x7f35d0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:00.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e934c700 1 -- 172.21.15.94:0/1249322074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d00051c0 msgr2=0x7f35d000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:00.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e934c700 1 --2- 172.21.15.94:0/1249322074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d00051c0 0x7f35d000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:00.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e934c700 1 -- 172.21.15.94:0/1249322074 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d0003570 con 0x7f35d0004810 2023-09-30T00:19:00.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35ea34e700 1 --2- 172.21.15.94:0/1249322074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d00051c0 0x7f35d000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:00.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35ea34e700 1 -- 172.21.15.94:0/1249322074 reap_dead start 2023-09-30T00:19:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.162+0000 7f35e934c700 1 --2- 172.21.15.94:0/1249322074 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35d0004810 0x7f35d0004c80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f35d400c3c0 tx=0x7f35d4001e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10eba42ef921d598 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.163+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d400cc90 con 0x7f35d0004810 2023-09-30T00:19:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.163+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d400cdf0 con 0x7f35d0004810 2023-09-30T00:19:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.163+0000 7f35e8b4b700 1 -- 172.21.15.94:0/1249322074 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4005b50 con 0x7f35d0004810 2023-09-30T00:19:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.163+0000 7f35ebdb1700 1 -- 172.21.15.94:0/1249322074 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35d0004810 msgr2=0x7f35d0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.163+0000 7f35ebdb1700 1 --2- 172.21.15.94:0/1249322074 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35d0004810 0x7f35d0004c80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f35d400c3c0 tx=0x7f35d4001e50 comp rx=0 tx=0).stop 2023-09-30T00:19:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.163+0000 7f35ebdb1700 1 -- 172.21.15.94:0/1249322074 shutdown_connections 2023-09-30T00:19:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.163+0000 7f35ebdb1700 1 --2- 172.21.15.94:0/1249322074 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35d0004810 0x7f35d0004c80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.163+0000 7f35ebdb1700 1 -- 172.21.15.94:0/1249322074 >> 172.21.15.94:0/1249322074 conn(0x7f35e4068730 msgr2=0x7f35e4068b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.163+0000 7f35ebdb1700 1 -- 172.21.15.94:0/1249322074 shutdown_connections 2023-09-30T00:19:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.163+0000 7f35ebdb1700 1 -- 172.21.15.94:0/1249322074 wait complete. 2023-09-30T00:19:00.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.164+0000 7f35ebdb1700 1 Processor -- start 2023-09-30T00:19:00.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.164+0000 7f35ebdb1700 1 -- start start 2023-09-30T00:19:00.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ebdb1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35d0004810 0x7f35e409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:00.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ebdb1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35e409c650 0x7f35e414c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:00.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ebdb1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35e409e790 0x7f35e40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:00.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ebdb1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e414ca10 con 0x7f35e409c650 2023-09-30T00:19:00.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ebdb1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e414ccd0 con 0x7f35d0004810 2023-09-30T00:19:00.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ebdb1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e409f820 con 0x7f35e409e790 2023-09-30T00:19:00.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35e934c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35e409c650 0x7f35e414c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:00.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ea34e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35e409e790 0x7f35e40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:00.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35e934c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35e409c650 0x7f35e414c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58720/0 (socket says 172.21.15.94:58720) 2023-09-30T00:19:00.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ea34e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35e409e790 0x7f35e40a2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:57980/0 (socket says 172.21.15.94:57980) 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35e934c700 1 -- 172.21.15.94:0/917550953 learned_addr learned my addr 172.21.15.94:0/917550953 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35e9b4d700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35d0004810 0x7f35e409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ea34e700 1 -- 172.21.15.94:0/917550953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35d0004810 msgr2=0x7f35e409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ea34e700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35d0004810 0x7f35e409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ea34e700 1 -- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35e409c650 msgr2=0x7f35e414c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ea34e700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35e409c650 0x7f35e414c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ea34e700 1 -- 172.21.15.94:0/917550953 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e414c670 con 0x7f35e409e790 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ea34e700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35e409e790 0x7f35e40a2c70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f35e40b35d0 tx=0x7f35dc007dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35d9ffb700 1 -- 172.21.15.94:0/917550953 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc011ea0 con 0x7f35e409e790 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35d9ffb700 1 -- 172.21.15.94:0/917550953 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc00fa30 con 0x7f35e409e790 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35d9ffb700 1 -- 172.21.15.94:0/917550953 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc00fdf0 con 0x7f35e409e790 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.165+0000 7f35ebdb1700 1 -- 172.21.15.94:0/917550953 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e40a1e70 con 0x7f35e409e790 2023-09-30T00:19:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.166+0000 7f35ebdb1700 1 -- 172.21.15.94:0/917550953 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e409ee40 con 0x7f35e409e790 2023-09-30T00:19:00.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.166+0000 7f35ebdb1700 1 -- 172.21.15.94:0/917550953 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e406fbd0 con 0x7f35e409e790 2023-09-30T00:19:00.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.167+0000 7f35d9ffb700 1 -- 172.21.15.94:0/917550953 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f35dc019030 con 0x7f35e409e790 2023-09-30T00:19:00.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.167+0000 7f35d9ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:00.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.168+0000 7f35d9ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:00.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.168+0000 7f35d9ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:00.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.168+0000 7f35d9ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:00.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.168+0000 7f35d9ffb700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f35c8098780 0x7f35c809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:00.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.168+0000 7f35e9b4d700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f35c8098780 0x7f35c809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:00.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.168+0000 7f35d9ffb700 1 -- 172.21.15.94:0/917550953 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f35dc0c31e0 con 0x7f35e409e790 2023-09-30T00:19:00.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.168+0000 7f35e9b4d700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f35c8098780 0x7f35c809ac30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f35e00094d0 tx=0x7f35e000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:00.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.171+0000 7f35d9ffb700 1 -- 172.21.15.94:0/917550953 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f35dc0908b0 con 0x7f35e409e790 2023-09-30T00:19:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.397+0000 7f35ebdb1700 1 -- 172.21.15.94:0/917550953 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f35e40a4b10 con 0x7f35e409e790 2023-09-30T00:19:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.397+0000 7f35d9ffb700 1 -- 172.21.15.94:0/917550953 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f35dc021020 con 0x7f35e409e790 2023-09-30T00:19:00.398 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:00.398 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:00.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 10 mgrc shutdown 2023-09-30T00:19:00.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 1 -- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f35c8098780 msgr2=0x7f35c809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:00.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f35c8098780 0x7f35c809ac30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f35e00094d0 tx=0x7f35e000b040 comp rx=0 tx=0).stop 2023-09-30T00:19:00.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 1 -- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35e409e790 msgr2=0x7f35e40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:00.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35e409e790 0x7f35e40a2c70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f35e40b35d0 tx=0x7f35dc007dc0 comp rx=0 tx=0).stop 2023-09-30T00:19:00.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 1 -- 172.21.15.94:0/917550953 shutdown_connections 2023-09-30T00:19:00.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f35c8098780 0x7f35c809ac30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:00.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f35e409e790 0x7f35e40a2c70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:00.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35d0004810 0x7f35e409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:00.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 1 --2- 172.21.15.94:0/917550953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35e409c650 0x7f35e414c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:00.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 1 -- 172.21.15.94:0/917550953 >> 172.21.15.94:0/917550953 conn(0x7f35e4068730 msgr2=0x7f35e4068b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:00.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 1 -- 172.21.15.94:0/917550953 shutdown_connections 2023-09-30T00:19:00.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:00.399+0000 7f35ebdb1700 1 -- 172.21.15.94:0/917550953 wait complete. 2023-09-30T00:19:03.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:03.538 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.537+0000 7fda85909700 1 Processor -- start 2023-09-30T00:19:03.538 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.538+0000 7fda85909700 1 -- start start 2023-09-30T00:19:03.544 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.538+0000 7fda85909700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fda80067b50 0x7fda8009e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:03.544 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.538+0000 7fda85909700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda80067f20 con 0x7fda800b57c0 2023-09-30T00:19:03.544 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.538+0000 7fda85909700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8009eda0 con 0x7fda80134790 2023-09-30T00:19:03.544 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.538+0000 7fda85909700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda80141a60 con 0x7fda80067b50 2023-09-30T00:19:03.544 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.538+0000 7fda7f7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fda800b57c0 0x7fda80148f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49732/0 (socket says 172.21.15.94:49732) 2023-09-30T00:19:03.544 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.538+0000 7fda7f7fe700 1 -- 172.21.15.94:0/2059558265 learned_addr learned my addr 172.21.15.94:0/2059558265 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:19:03.545 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.538+0000 7fda7effd700 1 --2- 172.21.15.94:0/2059558265 >> v2:172.21.15.121:3300/0 conn(0x7fda80067b50 0x7fda8009e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:03.545 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064295294 0 0) 0x7fda80067f20 con 0x7fda800b57c0 2023-09-30T00:19:03.545 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda6c003570 con 0x7fda800b57c0 2023-09-30T00:19:03.545 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475468889 0 0) 0x7fda8009eda0 con 0x7fda80134790 2023-09-30T00:19:03.545 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda80067f20 con 0x7fda80134790 2023-09-30T00:19:03.545 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1475297073 0 0) 0x7fda6c003570 con 0x7fda800b57c0 2023-09-30T00:19:03.545 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 >> v2:172.21.15.121:3300/0 conn(0x7fda80067b50 msgr2=0x7fda8009e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:03.545 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 --2- 172.21.15.94:0/2059558265 >> v2:172.21.15.121:3300/0 conn(0x7fda80067b50 0x7fda8009e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:03.545 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 >> v1:172.21.15.121:6789/0 conn(0x7fda80134790 legacy=0x7fda800b50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:03.545 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda800a5180 con 0x7fda800b57c0 2023-09-30T00:19:03.545 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3153706020 0 0) 0x7fda78001470 con 0x7fda800b57c0 2023-09-30T00:19:03.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 >> v1:172.21.15.94:6789/0 conn(0x7fda800b57c0 legacy=0x7fda80148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:03.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 --2- 172.21.15.94:0/2059558265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda6c003bf0 0x7fda6c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:03.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 --2- 172.21.15.94:0/2059558265 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fda6c004810 0x7fda6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:03.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 --2- 172.21.15.94:0/2059558265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda6c0051c0 0x7fda6c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:03.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80141a60 con 0x7fda6c003bf0 2023-09-30T00:19:03.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c00c0b0 con 0x7fda6c0051c0 2023-09-30T00:19:03.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c00c270 con 0x7fda6c004810 2023-09-30T00:19:03.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7effd700 1 --2- 172.21.15.94:0/2059558265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda6c003bf0 0x7fda6c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:03.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7e7fc700 1 --2- 172.21.15.94:0/2059558265 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fda6c004810 0x7fda6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:03.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7f7fe700 1 --2- 172.21.15.94:0/2059558265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda6c0051c0 0x7fda6c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:03.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7effd700 1 -- 172.21.15.94:0/2059558265 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fda6c004810 msgr2=0x7fda6c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:03.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7effd700 1 --2- 172.21.15.94:0/2059558265 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fda6c004810 0x7fda6c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:03.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7effd700 1 -- 172.21.15.94:0/2059558265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda6c0051c0 msgr2=0x7fda6c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:03.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7effd700 1 --2- 172.21.15.94:0/2059558265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda6c0051c0 0x7fda6c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:03.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7effd700 1 -- 172.21.15.94:0/2059558265 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c003570 con 0x7fda6c003bf0 2023-09-30T00:19:03.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7f7fe700 1 --2- 172.21.15.94:0/2059558265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda6c0051c0 0x7fda6c00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:19:03.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7f7fe700 1 -- 172.21.15.94:0/2059558265 reap_dead start 2023-09-30T00:19:03.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.539+0000 7fda7effd700 1 --2- 172.21.15.94:0/2059558265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda6c003bf0 0x7fda6c003fe0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fda7000e210 tx=0x7fda7000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49221df399319480 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:03.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.540+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000eac0 con 0x7fda6c003bf0 2023-09-30T00:19:03.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.540+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7000f040 con 0x7fda6c003bf0 2023-09-30T00:19:03.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.540+0000 7fda7dffb700 1 -- 172.21.15.94:0/2059558265 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000ee70 con 0x7fda6c003bf0 2023-09-30T00:19:03.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.540+0000 7fda85909700 1 -- 172.21.15.94:0/2059558265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda6c003bf0 msgr2=0x7fda6c003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:03.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.540+0000 7fda85909700 1 --2- 172.21.15.94:0/2059558265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda6c003bf0 0x7fda6c003fe0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fda7000e210 tx=0x7fda7000b040 comp rx=0 tx=0).stop 2023-09-30T00:19:03.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.540+0000 7fda85909700 1 -- 172.21.15.94:0/2059558265 shutdown_connections 2023-09-30T00:19:03.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.540+0000 7fda85909700 1 --2- 172.21.15.94:0/2059558265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda6c003bf0 0x7fda6c003fe0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:03.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.540+0000 7fda85909700 1 -- 172.21.15.94:0/2059558265 >> 172.21.15.94:0/2059558265 conn(0x7fda8006cb10 msgr2=0x7fda8006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:03.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.540+0000 7fda85909700 1 -- 172.21.15.94:0/2059558265 shutdown_connections 2023-09-30T00:19:03.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.540+0000 7fda85909700 1 -- 172.21.15.94:0/2059558265 wait complete. 2023-09-30T00:19:03.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.541+0000 7fda85909700 1 Processor -- start 2023-09-30T00:19:03.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.541+0000 7fda85909700 1 -- start start 2023-09-30T00:19:03.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda85909700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda6c003bf0 0x7fda8009b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:03.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda85909700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fda8009d7d0 0x7fda8009c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:03.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda85909700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda8013d860 0x7fda8013dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:03.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda85909700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8006a100 con 0x7fda8013d860 2023-09-30T00:19:03.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda85909700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8013e310 con 0x7fda6c003bf0 2023-09-30T00:19:03.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda85909700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8013e5b0 con 0x7fda8009d7d0 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda7e7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fda8009d7d0 0x7fda8009c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda7e7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fda8009d7d0 0x7fda8009c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:57992/0 (socket says 172.21.15.94:57992) 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda7e7fc700 1 -- 172.21.15.94:0/2299286683 learned_addr learned my addr 172.21.15.94:0/2299286683 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda7f7fe700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda8013d860 0x7fda8013dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda7effd700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda6c003bf0 0x7fda8009b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda7e7fc700 1 -- 172.21.15.94:0/2299286683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda6c003bf0 msgr2=0x7fda8009b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda7e7fc700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda6c003bf0 0x7fda8009b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda7e7fc700 1 -- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda8013d860 msgr2=0x7fda8013dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda7e7fc700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda8013d860 0x7fda8013dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda7e7fc700 1 -- 172.21.15.94:0/2299286683 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8009ce40 con 0x7fda8009d7d0 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda7f7fe700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda8013d860 0x7fda8013dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.542+0000 7fda7e7fc700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fda8009d7d0 0x7fda8009c900 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fda68002b80 tx=0x7fda68007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:03.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.543+0000 7fda76ffd700 1 -- 172.21.15.94:0/2299286683 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda68006a30 con 0x7fda8009d7d0 2023-09-30T00:19:03.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.543+0000 7fda76ffd700 1 -- 172.21.15.94:0/2299286683 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda68006b90 con 0x7fda8009d7d0 2023-09-30T00:19:03.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.543+0000 7fda76ffd700 1 -- 172.21.15.94:0/2299286683 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda68007050 con 0x7fda8009d7d0 2023-09-30T00:19:03.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.543+0000 7fda85909700 1 -- 172.21.15.94:0/2299286683 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8014fd80 con 0x7fda8009d7d0 2023-09-30T00:19:03.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.543+0000 7fda85909700 1 -- 172.21.15.94:0/2299286683 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda80150270 con 0x7fda8009d7d0 2023-09-30T00:19:03.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.543+0000 7fda85909700 1 -- 172.21.15.94:0/2299286683 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8009dfb0 con 0x7fda8009d7d0 2023-09-30T00:19:03.551 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.545+0000 7fda76ffd700 1 -- 172.21.15.94:0/2299286683 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fda68007480 con 0x7fda8009d7d0 2023-09-30T00:19:03.551 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.545+0000 7fda76ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:03.551 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.545+0000 7fda76ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:03.551 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.545+0000 7fda76ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:03.551 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.545+0000 7fda76ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:03.551 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.545+0000 7fda76ffd700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fda60098780 0x7fda6009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:03.551 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.545+0000 7fda76ffd700 1 -- 172.21.15.94:0/2299286683 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fda680c1300 con 0x7fda8009d7d0 2023-09-30T00:19:03.551 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.545+0000 7fda7effd700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fda60098780 0x7fda6009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:03.552 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.545+0000 7fda7effd700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fda60098780 0x7fda6009ac30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fda7000afd0 tx=0x7fda7001a000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:03.552 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.548+0000 7fda76ffd700 1 -- 172.21.15.94:0/2299286683 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fda6808e940 con 0x7fda8009d7d0 2023-09-30T00:19:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.772+0000 7fda85909700 1 -- 172.21.15.94:0/2299286683 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fda8009fa60 con 0x7fda8009d7d0 2023-09-30T00:19:03.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.773+0000 7fda76ffd700 1 -- 172.21.15.94:0/2299286683 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fda68007730 con 0x7fda8009d7d0 2023-09-30T00:19:03.773 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:03.773 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.774+0000 7fda85909700 10 mgrc shutdown 2023-09-30T00:19:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.774+0000 7fda85909700 1 -- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fda60098780 msgr2=0x7fda6009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.774+0000 7fda85909700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fda60098780 0x7fda6009ac30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fda7000afd0 tx=0x7fda7001a000 comp rx=0 tx=0).stop 2023-09-30T00:19:03.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.774+0000 7fda85909700 1 -- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fda8009d7d0 msgr2=0x7fda8009c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:03.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.774+0000 7fda85909700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fda8009d7d0 0x7fda8009c900 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fda68002b80 tx=0x7fda68007d30 comp rx=0 tx=0).stop 2023-09-30T00:19:03.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.774+0000 7fda85909700 1 -- 172.21.15.94:0/2299286683 shutdown_connections 2023-09-30T00:19:03.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.774+0000 7fda85909700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fda60098780 0x7fda6009ac30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:03.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.774+0000 7fda85909700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda8013d860 0x7fda8013dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:03.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.774+0000 7fda85909700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda6c003bf0 0x7fda8009b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:03.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.774+0000 7fda85909700 1 --2- 172.21.15.94:0/2299286683 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fda8009d7d0 0x7fda8009c900 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:03.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.775+0000 7fda85909700 1 -- 172.21.15.94:0/2299286683 >> 172.21.15.94:0/2299286683 conn(0x7fda8006cb10 msgr2=0x7fda8009e290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:03.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.775+0000 7fda85909700 1 -- 172.21.15.94:0/2299286683 shutdown_connections 2023-09-30T00:19:03.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:03.775+0000 7fda85909700 1 -- 172.21.15.94:0/2299286683 wait complete. 2023-09-30T00:19:06.788 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:06.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.913+0000 7f9bbd903700 1 Processor -- start 2023-09-30T00:19:06.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.913+0000 7f9bbd903700 1 -- start start 2023-09-30T00:19:06.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bbd903700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9bb80b7160 0x7f9bb80a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:06.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bbd903700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb80a1310 con 0x7f9bb813fd70 2023-09-30T00:19:06.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bbd903700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb8071020 con 0x7f9bb8070c50 2023-09-30T00:19:06.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bbd903700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb80b76a0 con 0x7f9bb80b7160 2023-09-30T00:19:06.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bae7fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9bb80b7160 0x7f9bb80a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:06.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bb77fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9bb813fd70 0x7f9bb814b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53934/0 (socket says 172.21.15.94:53934) 2023-09-30T00:19:06.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bb77fe700 1 -- 172.21.15.94:0/3426306165 learned_addr learned my addr 172.21.15.94:0/3426306165 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:19:06.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bae7fc700 1 -- 172.21.15.94:0/3426306165 >> v1:172.21.15.121:6789/0 conn(0x7f9bb8070c50 legacy=0x7f9bb80a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:06.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bae7fc700 1 -- 172.21.15.94:0/3426306165 >> v1:172.21.15.94:6789/0 conn(0x7f9bb813fd70 legacy=0x7f9bb814b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:06.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bb67fc700 1 -- 172.21.15.94:0/3426306165 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049634236 0 0) 0x7f9bb0001490 con 0x7f9bb813fd70 2023-09-30T00:19:06.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bae7fc700 1 -- 172.21.15.94:0/3426306165 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb80b7940 con 0x7f9bb80b7160 2023-09-30T00:19:06.918 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bae7fc700 1 --2- 172.21.15.94:0/3426306165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb80b7160 0x7f9bb80a0dd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00098e0 tx=0x7f9ba00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f99a0f7c004987 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:06.918 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bb67fc700 1 -- 172.21.15.94:0/3426306165 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0009e80 con 0x7f9bb80b7160 2023-09-30T00:19:06.918 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bb67fc700 1 -- 172.21.15.94:0/3426306165 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba000b8f0 con 0x7f9bb80b7160 2023-09-30T00:19:06.918 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.914+0000 7f9bb67fc700 1 -- 172.21.15.94:0/3426306165 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba000bc70 con 0x7f9bb80b7160 2023-09-30T00:19:06.918 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.915+0000 7f9bbd903700 1 -- 172.21.15.94:0/3426306165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb80b7160 msgr2=0x7f9bb80a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:06.918 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.915+0000 7f9bbd903700 1 --2- 172.21.15.94:0/3426306165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb80b7160 0x7f9bb80a0dd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00098e0 tx=0x7f9ba00091a0 comp rx=0 tx=0).stop 2023-09-30T00:19:06.918 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.915+0000 7f9bbd903700 1 -- 172.21.15.94:0/3426306165 shutdown_connections 2023-09-30T00:19:06.919 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.915+0000 7f9bbd903700 1 --2- 172.21.15.94:0/3426306165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb80b7160 0x7f9bb80a0dd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:06.919 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.915+0000 7f9bbd903700 1 -- 172.21.15.94:0/3426306165 >> 172.21.15.94:0/3426306165 conn(0x7f9bb806a090 msgr2=0x7f9bb80702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:06.919 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.915+0000 7f9bbd903700 1 -- 172.21.15.94:0/3426306165 shutdown_connections 2023-09-30T00:19:06.919 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.915+0000 7f9bbd903700 1 -- 172.21.15.94:0/3426306165 wait complete. 2023-09-30T00:19:06.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.916+0000 7f9bbd903700 1 Processor -- start 2023-09-30T00:19:06.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.916+0000 7f9bbd903700 1 -- start start 2023-09-30T00:19:06.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bbd903700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bb8070c50 0x7f9bb80ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:06.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bbd903700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb80b7160 0x7f9bb80a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:06.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bbd903700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9bb813fd70 0x7f9bb814ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:06.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bbd903700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb80aacc0 con 0x7f9bb80b7160 2023-09-30T00:19:06.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bbd903700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb8141840 con 0x7f9bb8070c50 2023-09-30T00:19:06.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bbd903700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb80a4520 con 0x7f9bb813fd70 2023-09-30T00:19:06.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bae7fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb80b7160 0x7f9bb80a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:06.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bae7fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb80b7160 0x7f9bb80a3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52738/0 (socket says 172.21.15.94:52738) 2023-09-30T00:19:06.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bae7fc700 1 -- 172.21.15.94:0/2847484688 learned_addr learned my addr 172.21.15.94:0/2847484688 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:06.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bb77fe700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9bb813fd70 0x7f9bb814ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:06.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bb6ffd700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bb8070c50 0x7f9bb80ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:06.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bae7fc700 1 -- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9bb813fd70 msgr2=0x7f9bb814ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:06.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bae7fc700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9bb813fd70 0x7f9bb814ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:06.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bae7fc700 1 -- 172.21.15.94:0/2847484688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bb8070c50 msgr2=0x7f9bb80ab430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:06.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bae7fc700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bb8070c50 0x7f9bb80ab430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:06.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bae7fc700 1 -- 172.21.15.94:0/2847484688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba0009420 con 0x7f9bb80b7160 2023-09-30T00:19:06.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bae7fc700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb80b7160 0x7f9bb80a3c40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9ba000b3c0 tx=0x7f9ba0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:06.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9baf7fe700 1 -- 172.21.15.94:0/2847484688 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba001f070 con 0x7f9bb80b7160 2023-09-30T00:19:06.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9baf7fe700 1 -- 172.21.15.94:0/2847484688 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba0000c90 con 0x7f9bb80b7160 2023-09-30T00:19:06.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9bbd903700 1 -- 172.21.15.94:0/2847484688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb80a47e0 con 0x7f9bb80b7160 2023-09-30T00:19:06.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.917+0000 7f9baf7fe700 1 -- 172.21.15.94:0/2847484688 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0025420 con 0x7f9bb80b7160 2023-09-30T00:19:06.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.918+0000 7f9bbd903700 1 -- 172.21.15.94:0/2847484688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb80a6ae0 con 0x7f9bb80b7160 2023-09-30T00:19:06.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.918+0000 7f9bbd903700 1 -- 172.21.15.94:0/2847484688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb80707c0 con 0x7f9bb80b7160 2023-09-30T00:19:06.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.919+0000 7f9baf7fe700 1 -- 172.21.15.94:0/2847484688 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f9ba0023070 con 0x7f9bb80b7160 2023-09-30T00:19:06.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.919+0000 7f9baf7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:06.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.920+0000 7f9baf7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:06.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.920+0000 7f9baf7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:06.926 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.920+0000 7f9baf7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:06.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.920+0000 7f9baf7fe700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9b98098780 0x7f9b9809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:06.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.920+0000 7f9bb6ffd700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9b98098780 0x7f9b9809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:06.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.920+0000 7f9baf7fe700 1 -- 172.21.15.94:0/2847484688 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f9ba00c19a0 con 0x7f9bb80b7160 2023-09-30T00:19:06.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.920+0000 7f9bb6ffd700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9b98098780 0x7f9b9809ac30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8003ac0 tx=0x7f9ba8007ea0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:06.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:06.924+0000 7f9baf7fe700 1 -- 172.21.15.94:0/2847484688 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ba008f070 con 0x7f9bb80b7160 2023-09-30T00:19:07.145 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.145+0000 7f9bbd903700 1 -- 172.21.15.94:0/2847484688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9bb8071630 con 0x7f9bb80b7160 2023-09-30T00:19:07.146 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.145+0000 7f9baf7fe700 1 -- 172.21.15.94:0/2847484688 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9ba0015080 con 0x7f9bb80b7160 2023-09-30T00:19:07.146 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:07.146 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:07.148 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 10 mgrc shutdown 2023-09-30T00:19:07.148 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 1 -- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9b98098780 msgr2=0x7f9b9809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:07.148 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9b98098780 0x7f9b9809ac30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8003ac0 tx=0x7f9ba8007ea0 comp rx=0 tx=0).stop 2023-09-30T00:19:07.148 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 1 -- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb80b7160 msgr2=0x7f9bb80a3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:07.148 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb80b7160 0x7f9bb80a3c40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9ba000b3c0 tx=0x7f9ba0017000 comp rx=0 tx=0).stop 2023-09-30T00:19:07.148 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 1 -- 172.21.15.94:0/2847484688 shutdown_connections 2023-09-30T00:19:07.149 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9b98098780 0x7f9b9809ac30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:07.149 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9bb813fd70 0x7f9bb814ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:07.149 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bb8070c50 0x7f9bb80ab430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:07.149 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 1 --2- 172.21.15.94:0/2847484688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb80b7160 0x7f9bb80a3c40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:07.149 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 1 -- 172.21.15.94:0/2847484688 >> 172.21.15.94:0/2847484688 conn(0x7f9bb806a090 msgr2=0x7f9bb8067cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:07.149 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 1 -- 172.21.15.94:0/2847484688 shutdown_connections 2023-09-30T00:19:07.149 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:07.147+0000 7f9bbd903700 1 -- 172.21.15.94:0/2847484688 wait complete. 2023-09-30T00:19:09.132 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:19:09.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:19:10.160 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:10.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.283+0000 7fc69e79c700 1 Processor -- start 2023-09-30T00:19:10.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.283+0000 7fc69e79c700 1 -- start start 2023-09-30T00:19:10.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.283+0000 7fc69e79c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc698067e30 0x7fc6980a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:10.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.283+0000 7fc69e79c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc6980a94d0 0x7fc69814b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:10.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.283+0000 7fc69e79c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc698068200 con 0x7fc6980a9ea0 2023-09-30T00:19:10.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.283+0000 7fc69e79c700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6980a22b0 con 0x7fc698067e30 2023-09-30T00:19:10.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.283+0000 7fc69e79c700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6980a24a0 con 0x7fc6980a94d0 2023-09-30T00:19:10.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.283+0000 7fc697fff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc698067e30 0x7fc6980a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:10.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc68ffff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc6980a9ea0 0x7fc69813fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53944/0 (socket says 172.21.15.94:53944) 2023-09-30T00:19:10.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc697fff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc698067e30 0x7fc6980a1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52750/0 (socket says 172.21.15.94:52750) 2023-09-30T00:19:10.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc68ffff700 1 -- 172.21.15.94:0/3716211400 learned_addr learned my addr 172.21.15.94:0/3716211400 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:19:10.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc69cd39700 1 --2- 172.21.15.94:0/3716211400 >> v2:172.21.15.121:3300/0 conn(0x7fc6980a94d0 0x7fc69814b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc697fff700 1 -- 172.21.15.94:0/3716211400 >> v2:172.21.15.121:3300/0 conn(0x7fc6980a94d0 msgr2=0x7fc69814b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc697fff700 1 --2- 172.21.15.94:0/3716211400 >> v2:172.21.15.121:3300/0 conn(0x7fc6980a94d0 0x7fc69814b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc697fff700 1 -- 172.21.15.94:0/3716211400 >> v1:172.21.15.94:6789/0 conn(0x7fc6980a9ea0 legacy=0x7fc69813fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc697fff700 1 -- 172.21.15.94:0/3716211400 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6980a2740 con 0x7fc698067e30 2023-09-30T00:19:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc6977fe700 1 -- 172.21.15.94:0/3716211400 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542869205 0 0) 0x7fc698068200 con 0x7fc6980a9ea0 2023-09-30T00:19:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc697fff700 1 --2- 172.21.15.94:0/3716211400 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc698067e30 0x7fc6980a1d70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc6980a2ea0 tx=0x7fc6880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3208883cd42f4f41 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc6977fe700 1 -- 172.21.15.94:0/3716211400 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688011630 con 0x7fc698067e30 2023-09-30T00:19:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc6977fe700 1 -- 172.21.15.94:0/3716211400 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc688011790 con 0x7fc698067e30 2023-09-30T00:19:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.284+0000 7fc6977fe700 1 -- 172.21.15.94:0/3716211400 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688011b30 con 0x7fc698067e30 2023-09-30T00:19:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.285+0000 7fc69e79c700 1 -- 172.21.15.94:0/3716211400 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc698067e30 msgr2=0x7fc6980a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.285+0000 7fc69e79c700 1 --2- 172.21.15.94:0/3716211400 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc698067e30 0x7fc6980a1d70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc6980a2ea0 tx=0x7fc6880091a0 comp rx=0 tx=0).stop 2023-09-30T00:19:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.285+0000 7fc69e79c700 1 -- 172.21.15.94:0/3716211400 shutdown_connections 2023-09-30T00:19:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.285+0000 7fc69e79c700 1 --2- 172.21.15.94:0/3716211400 >> v2:172.21.15.121:3300/0 conn(0x7fc6980a94d0 0x7fc69814b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.285+0000 7fc69e79c700 1 --2- 172.21.15.94:0/3716211400 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc698067e30 0x7fc6980a1d70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.285+0000 7fc69e79c700 1 -- 172.21.15.94:0/3716211400 >> 172.21.15.94:0/3716211400 conn(0x7fc6980702f0 msgr2=0x7fc6980706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.285+0000 7fc69e79c700 1 -- 172.21.15.94:0/3716211400 shutdown_connections 2023-09-30T00:19:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.285+0000 7fc69e79c700 1 -- 172.21.15.94:0/3716211400 wait complete. 2023-09-30T00:19:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc69e79c700 1 Processor -- start 2023-09-30T00:19:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc69e79c700 1 -- start start 2023-09-30T00:19:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc69e79c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc698067e30 0x7fc698144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc69e79c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6980a94d0 0x7fc698141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc69e79c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc698142000 0x7fc6981537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc69e79c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6981431b0 con 0x7fc6980a94d0 2023-09-30T00:19:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc69e79c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc698143620 con 0x7fc698142000 2023-09-30T00:19:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc69e79c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc698143900 con 0x7fc698067e30 2023-09-30T00:19:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc697fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc698067e30 0x7fc698144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc68ffff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6980a94d0 0x7fc698141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc697fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc698067e30 0x7fc698144070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:33436/0 (socket says 172.21.15.94:33436) 2023-09-30T00:19:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc697fff700 1 -- 172.21.15.94:0/2570565692 learned_addr learned my addr 172.21.15.94:0/2570565692 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.286+0000 7fc69cd39700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc698142000 0x7fc6981537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.287+0000 7fc697fff700 1 -- 172.21.15.94:0/2570565692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc698142000 msgr2=0x7fc6981537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.287+0000 7fc697fff700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc698142000 0x7fc6981537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.287+0000 7fc697fff700 1 -- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6980a94d0 msgr2=0x7fc698141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.287+0000 7fc697fff700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6980a94d0 0x7fc698141ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.287+0000 7fc697fff700 1 -- 172.21.15.94:0/2570565692 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc688009420 con 0x7fc698067e30 2023-09-30T00:19:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.287+0000 7fc68ffff700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6980a94d0 0x7fc698141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.287+0000 7fc697fff700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc698067e30 0x7fc698144070 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc6880091e0 tx=0x7fc688017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.287+0000 7fc694ff9700 1 -- 172.21.15.94:0/2570565692 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688000d10 con 0x7fc698067e30 2023-09-30T00:19:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.287+0000 7fc694ff9700 1 -- 172.21.15.94:0/2570565692 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc688011b10 con 0x7fc698067e30 2023-09-30T00:19:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.287+0000 7fc69e79c700 1 -- 172.21.15.94:0/2570565692 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc698142620 con 0x7fc698067e30 2023-09-30T00:19:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.287+0000 7fc694ff9700 1 -- 172.21.15.94:0/2570565692 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688025660 con 0x7fc698067e30 2023-09-30T00:19:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.287+0000 7fc69e79c700 1 -- 172.21.15.94:0/2570565692 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6980a5860 con 0x7fc698067e30 2023-09-30T00:19:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.288+0000 7fc69e79c700 1 -- 172.21.15.94:0/2570565692 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6980a59f0 con 0x7fc698067e30 2023-09-30T00:19:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.289+0000 7fc694ff9700 1 -- 172.21.15.94:0/2570565692 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc688025a90 con 0x7fc698067e30 2023-09-30T00:19:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.289+0000 7fc694ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.289+0000 7fc694ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.289+0000 7fc694ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.289+0000 7fc694ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.289+0000 7fc694ff9700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc6780987b0 0x7fc67809ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.290+0000 7fc68ffff700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc6780987b0 0x7fc67809ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.290+0000 7fc694ff9700 1 -- 172.21.15.94:0/2570565692 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fc6880c1fe0 con 0x7fc698067e30 2023-09-30T00:19:10.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.290+0000 7fc68ffff700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc6780987b0 0x7fc67809ac60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc698070d40 tx=0x7fc680007f50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:10.294 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.293+0000 7fc694ff9700 1 -- 172.21.15.94:0/2570565692 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc68808f5e0 con 0x7fc698067e30 2023-09-30T00:19:10.515 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.515+0000 7fc69e79c700 1 -- 172.21.15.94:0/2570565692 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc698143a40 con 0x7fc698067e30 2023-09-30T00:19:10.515 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.515+0000 7fc694ff9700 1 -- 172.21.15.94:0/2570565692 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc698143a40 con 0x7fc698067e30 2023-09-30T00:19:10.516 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:10.516 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:10.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 10 mgrc shutdown 2023-09-30T00:19:10.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 1 -- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc6780987b0 msgr2=0x7fc67809ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:10.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc6780987b0 0x7fc67809ac60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc698070d40 tx=0x7fc680007f50 comp rx=0 tx=0).stop 2023-09-30T00:19:10.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 1 -- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc698067e30 msgr2=0x7fc698144070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:10.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc698067e30 0x7fc698144070 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc6880091e0 tx=0x7fc688017000 comp rx=0 tx=0).stop 2023-09-30T00:19:10.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 1 -- 172.21.15.94:0/2570565692 shutdown_connections 2023-09-30T00:19:10.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc6780987b0 0x7fc67809ac60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:10.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc698067e30 0x7fc698144070 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:10.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc698142000 0x7fc6981537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:10.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 1 --2- 172.21.15.94:0/2570565692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6980a94d0 0x7fc698141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:10.519 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 1 -- 172.21.15.94:0/2570565692 >> 172.21.15.94:0/2570565692 conn(0x7fc6980702f0 msgr2=0x7fc6980b7580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:10.519 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 1 -- 172.21.15.94:0/2570565692 shutdown_connections 2023-09-30T00:19:10.519 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:10.517+0000 7fc69e79c700 1 -- 172.21.15.94:0/2570565692 wait complete. 2023-09-30T00:19:13.530 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:13.656 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.655+0000 7f7b5d73b700 1 Processor -- start 2023-09-30T00:19:13.656 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.655+0000 7f7b5d73b700 1 -- start start 2023-09-30T00:19:13.658 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.656+0000 7f7b5d73b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7b580a4470 0x7f7b580b5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:13.658 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.656+0000 7f7b5d73b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a4840 0x7f7b580a5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:13.658 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.656+0000 7f7b5d73b700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b580a6050 con 0x7f7b580a6fc0 2023-09-30T00:19:13.658 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.656+0000 7f7b5d73b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b580b4dd0 con 0x7f7b580a4470 2023-09-30T00:19:13.659 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.656+0000 7f7b5d73b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5806ea50 con 0x7f7b580a4840 2023-09-30T00:19:13.659 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.656+0000 7f7b567fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a4840 0x7f7b580a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:13.659 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.656+0000 7f7b567fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a4840 0x7f7b580a5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:33442/0 (socket says 172.21.15.94:33442) 2023-09-30T00:19:13.659 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.656+0000 7f7b567fc700 1 -- 172.21.15.94:0/1144964532 learned_addr learned my addr 172.21.15.94:0/1144964532 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:13.659 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.656+0000 7f7b56ffd700 1 --2- 172.21.15.94:0/1144964532 >> v2:172.21.15.94:3300/0 conn(0x7f7b580a4470 0x7f7b580b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:13.659 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.656+0000 7f7b567fc700 1 -- 172.21.15.94:0/1144964532 >> v2:172.21.15.94:3300/0 conn(0x7f7b580a4470 msgr2=0x7f7b580b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:13.659 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.656+0000 7f7b567fc700 1 --2- 172.21.15.94:0/1144964532 >> v2:172.21.15.94:3300/0 conn(0x7f7b580a4470 0x7f7b580b5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:13.659 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.656+0000 7f7b567fc700 1 -- 172.21.15.94:0/1144964532 >> v1:172.21.15.121:6789/0 conn(0x7f7b580a6fc0 legacy=0x7f7b58141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:13.659 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b567fc700 1 -- 172.21.15.94:0/1144964532 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b580b5a90 con 0x7f7b580a4840 2023-09-30T00:19:13.659 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b56ffd700 1 --2- 172.21.15.94:0/1144964532 >> v2:172.21.15.94:3300/0 conn(0x7f7b580a4470 0x7f7b580b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:13.659 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b567fc700 1 --2- 172.21.15.94:0/1144964532 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a4840 0x7f7b580a5720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7b400099b0 tx=0x7f7b40009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad33da3699a9b61b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b55ffb700 1 -- 172.21.15.94:0/1144964532 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40009ea0 con 0x7f7b580a4840 2023-09-30T00:19:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b55ffb700 1 -- 172.21.15.94:0/1144964532 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4000b860 con 0x7f7b580a4840 2023-09-30T00:19:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b55ffb700 1 -- 172.21.15.94:0/1144964532 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4000bc00 con 0x7f7b580a4840 2023-09-30T00:19:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1144964532 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a4840 msgr2=0x7f7b580a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b5d73b700 1 --2- 172.21.15.94:0/1144964532 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a4840 0x7f7b580a5720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7b400099b0 tx=0x7f7b40009250 comp rx=0 tx=0).stop 2023-09-30T00:19:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1144964532 shutdown_connections 2023-09-30T00:19:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b5d73b700 1 --2- 172.21.15.94:0/1144964532 >> v2:172.21.15.94:3300/0 conn(0x7f7b580a4470 0x7f7b580b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b5d73b700 1 --2- 172.21.15.94:0/1144964532 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a4840 0x7f7b580a5720 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1144964532 >> 172.21.15.94:0/1144964532 conn(0x7f7b5806cbf0 msgr2=0x7f7b5806cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1144964532 shutdown_connections 2023-09-30T00:19:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.657+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1144964532 wait complete. 2023-09-30T00:19:13.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.658+0000 7f7b5d73b700 1 Processor -- start 2023-09-30T00:19:13.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b5d73b700 1 -- start start 2023-09-30T00:19:13.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b5d73b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b580a4470 0x7f7b58154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:13.663 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b5d73b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b580a4840 0x7f7b580b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b5d73b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a6fc0 0x7f7b58145f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b5d73b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b580afec0 con 0x7f7b580a4470 2023-09-30T00:19:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b5d73b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58068b70 con 0x7f7b580a4840 2023-09-30T00:19:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b5d73b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58068e30 con 0x7f7b580a6fc0 2023-09-30T00:19:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b56ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b580a4470 0x7f7b58154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b56ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b580a4470 0x7f7b58154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52774/0 (socket says 172.21.15.94:52774) 2023-09-30T00:19:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b56ffd700 1 -- 172.21.15.94:0/1128010715 learned_addr learned my addr 172.21.15.94:0/1128010715 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b577fe700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a6fc0 0x7f7b58145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b567fc700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b580a4840 0x7f7b580b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b56ffd700 1 -- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a6fc0 msgr2=0x7f7b58145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b56ffd700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a6fc0 0x7f7b58145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:13.664 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b56ffd700 1 -- 172.21.15.94:0/1128010715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b580a4840 msgr2=0x7f7b580b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:13.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.659+0000 7f7b56ffd700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b580a4840 0x7f7b580b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:13.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.660+0000 7f7b56ffd700 1 -- 172.21.15.94:0/1128010715 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b48009a20 con 0x7f7b580a4470 2023-09-30T00:19:13.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.660+0000 7f7b577fe700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a6fc0 0x7f7b58145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:13.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.660+0000 7f7b56ffd700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b580a4470 0x7f7b58154230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800c0c0 tx=0x7f7b4800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:13.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.660+0000 7f7b4effd700 1 -- 172.21.15.94:0/1128010715 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4800cc00 con 0x7f7b580a4470 2023-09-30T00:19:13.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.660+0000 7f7b4effd700 1 -- 172.21.15.94:0/1128010715 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4800f040 con 0x7f7b580a4470 2023-09-30T00:19:13.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.660+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1128010715 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b400094f0 con 0x7f7b580a4470 2023-09-30T00:19:13.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.660+0000 7f7b4effd700 1 -- 172.21.15.94:0/1128010715 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48014480 con 0x7f7b580a4470 2023-09-30T00:19:13.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.660+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1128010715 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b58068730 con 0x7f7b580a4470 2023-09-30T00:19:13.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.661+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1128010715 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b58154080 con 0x7f7b580a4470 2023-09-30T00:19:13.665 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.662+0000 7f7b4effd700 1 -- 172.21.15.94:0/1128010715 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f7b480147f0 con 0x7f7b580a4470 2023-09-30T00:19:13.666 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.662+0000 7f7b4effd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:13.666 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.662+0000 7f7b4effd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:13.666 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.662+0000 7f7b4effd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:13.666 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.662+0000 7f7b4effd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:13.666 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.662+0000 7f7b4effd700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b38098780 0x7f7b3809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:13.666 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.662+0000 7f7b567fc700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b38098780 0x7f7b3809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:13.666 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.662+0000 7f7b4effd700 1 -- 172.21.15.94:0/1128010715 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f7b480c1620 con 0x7f7b580a4470 2023-09-30T00:19:13.666 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.663+0000 7f7b567fc700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b38098780 0x7f7b3809ac30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7b400099b0 tx=0x7f7b40017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:13.666 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.666+0000 7f7b4effd700 1 -- 172.21.15.94:0/1128010715 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b480545a0 con 0x7f7b580a4470 2023-09-30T00:19:13.888 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.887+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1128010715 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7b5806fbd0 con 0x7f7b580a4470 2023-09-30T00:19:13.888 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.887+0000 7f7b4effd700 1 -- 172.21.15.94:0/1128010715 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7b48092780 con 0x7f7b580a4470 2023-09-30T00:19:13.888 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:13.888 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:13.890 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.889+0000 7f7b5d73b700 10 mgrc shutdown 2023-09-30T00:19:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.890+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b38098780 msgr2=0x7f7b3809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.890+0000 7f7b5d73b700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b38098780 0x7f7b3809ac30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7b400099b0 tx=0x7f7b40017000 comp rx=0 tx=0).stop 2023-09-30T00:19:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.890+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b580a4470 msgr2=0x7f7b58154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.890+0000 7f7b5d73b700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b580a4470 0x7f7b58154230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800c0c0 tx=0x7f7b4800b040 comp rx=0 tx=0).stop 2023-09-30T00:19:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.890+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1128010715 shutdown_connections 2023-09-30T00:19:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.890+0000 7f7b5d73b700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b38098780 0x7f7b3809ac30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.890+0000 7f7b5d73b700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b580a6fc0 0x7f7b58145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:13.891 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.890+0000 7f7b5d73b700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b580a4470 0x7f7b58154230 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.890+0000 7f7b5d73b700 1 --2- 172.21.15.94:0/1128010715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b580a4840 0x7f7b580b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.890+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1128010715 >> 172.21.15.94:0/1128010715 conn(0x7f7b5806cbf0 msgr2=0x7f7b5806cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.890+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1128010715 shutdown_connections 2023-09-30T00:19:13.892 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:13.890+0000 7f7b5d73b700 1 -- 172.21.15.94:0/1128010715 wait complete. 2023-09-30T00:19:16.903 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:17.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.027+0000 7fea530a3700 1 Processor -- start 2023-09-30T00:19:17.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.028+0000 7fea530a3700 1 -- start start 2023-09-30T00:19:17.030 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.028+0000 7fea530a3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c070c50 0x7fea4c0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:17.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.028+0000 7fea530a3700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c0a1310 con 0x7fea4c0b7160 2023-09-30T00:19:17.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.028+0000 7fea530a3700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c071020 con 0x7fea4c13fd70 2023-09-30T00:19:17.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.028+0000 7fea530a3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c0b76a0 con 0x7fea4c070c50 2023-09-30T00:19:17.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.028+0000 7fea50e3f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c070c50 0x7fea4c0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:17.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.028+0000 7fea50e3f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c070c50 0x7fea4c0a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44630/0 (socket says 172.21.15.94:44630) 2023-09-30T00:19:17.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.028+0000 7fea43fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fea4c0b7160 0x7fea4c0a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38662/0 (socket says 172.21.15.94:38662) 2023-09-30T00:19:17.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.028+0000 7fea50e3f700 1 -- 172.21.15.94:0/950694068 learned_addr learned my addr 172.21.15.94:0/950694068 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:17.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea437fe700 1 -- 172.21.15.94:0/950694068 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006556429 0 0) 0x7fea3c001110 con 0x7fea4c0b7160 2023-09-30T00:19:17.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea50e3f700 1 -- 172.21.15.94:0/950694068 >> v1:172.21.15.121:6789/0 conn(0x7fea4c13fd70 legacy=0x7fea4c14b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:17.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea50e3f700 1 -- 172.21.15.94:0/950694068 >> v1:172.21.15.94:6789/0 conn(0x7fea4c0b7160 legacy=0x7fea4c0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:17.031 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea50e3f700 1 -- 172.21.15.94:0/950694068 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c0b7940 con 0x7fea4c070c50 2023-09-30T00:19:17.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea50e3f700 1 --2- 172.21.15.94:0/950694068 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c070c50 0x7fea4c0a0dd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fea480099e0 tx=0x7fea48009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=634f9ff5f3e0cd39 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:17.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea437fe700 1 -- 172.21.15.94:0/950694068 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4800b8f0 con 0x7fea4c070c50 2023-09-30T00:19:17.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea437fe700 1 -- 172.21.15.94:0/950694068 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4800ba50 con 0x7fea4c070c50 2023-09-30T00:19:17.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea437fe700 1 -- 172.21.15.94:0/950694068 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4800bdf0 con 0x7fea4c070c50 2023-09-30T00:19:17.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea530a3700 1 -- 172.21.15.94:0/950694068 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c070c50 msgr2=0x7fea4c0a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:17.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea530a3700 1 --2- 172.21.15.94:0/950694068 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c070c50 0x7fea4c0a0dd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fea480099e0 tx=0x7fea48009250 comp rx=0 tx=0).stop 2023-09-30T00:19:17.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea530a3700 1 -- 172.21.15.94:0/950694068 shutdown_connections 2023-09-30T00:19:17.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea530a3700 1 --2- 172.21.15.94:0/950694068 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c070c50 0x7fea4c0a0dd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:17.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.029+0000 7fea530a3700 1 -- 172.21.15.94:0/950694068 >> 172.21.15.94:0/950694068 conn(0x7fea4c06a090 msgr2=0x7fea4c0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:17.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.030+0000 7fea530a3700 1 -- 172.21.15.94:0/950694068 shutdown_connections 2023-09-30T00:19:17.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.030+0000 7fea530a3700 1 -- 172.21.15.94:0/950694068 wait complete. 2023-09-30T00:19:17.033 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.030+0000 7fea530a3700 1 Processor -- start 2023-09-30T00:19:17.035 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.031+0000 7fea530a3700 1 -- start start 2023-09-30T00:19:17.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.031+0000 7fea530a3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea4c070c50 0x7fea4c146cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:17.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.031+0000 7fea530a3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea4c0b7160 0x7fea4c140a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:17.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.031+0000 7fea530a3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c13fd70 0x7fea4c147df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:17.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.031+0000 7fea530a3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c13cc20 con 0x7fea4c070c50 2023-09-30T00:19:17.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.031+0000 7fea530a3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c1468c0 con 0x7fea4c0b7160 2023-09-30T00:19:17.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.031+0000 7fea530a3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c1484f0 con 0x7fea4c13fd70 2023-09-30T00:19:17.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.031+0000 7fea51640700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c13fd70 0x7fea4c147df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:17.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.031+0000 7fea50e3f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea4c070c50 0x7fea4c146cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:17.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.031+0000 7fea51640700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c13fd70 0x7fea4c147df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44640/0 (socket says 172.21.15.94:44640) 2023-09-30T00:19:17.036 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.031+0000 7fea51640700 1 -- 172.21.15.94:0/1679375635 learned_addr learned my addr 172.21.15.94:0/1679375635 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:17.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.031+0000 7fea43fff700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea4c0b7160 0x7fea4c140a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:17.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.032+0000 7fea51640700 1 -- 172.21.15.94:0/1679375635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea4c0b7160 msgr2=0x7fea4c140a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:17.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.032+0000 7fea51640700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea4c0b7160 0x7fea4c140a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:17.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.032+0000 7fea51640700 1 -- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea4c070c50 msgr2=0x7fea4c146cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:17.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.032+0000 7fea51640700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea4c070c50 0x7fea4c146cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:17.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.032+0000 7fea51640700 1 -- 172.21.15.94:0/1679375635 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea480094f0 con 0x7fea4c13fd70 2023-09-30T00:19:17.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.032+0000 7fea50e3f700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea4c070c50 0x7fea4c146cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:17.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.032+0000 7fea51640700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c13fd70 0x7fea4c147df0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fea4400c840 tx=0x7fea44001670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:17.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.032+0000 7fea40ff9700 1 -- 172.21.15.94:0/1679375635 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea44012630 con 0x7fea4c13fd70 2023-09-30T00:19:17.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.032+0000 7fea40ff9700 1 -- 172.21.15.94:0/1679375635 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea440127b0 con 0x7fea4c13fd70 2023-09-30T00:19:17.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.032+0000 7fea530a3700 1 -- 172.21.15.94:0/1679375635 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4c1487b0 con 0x7fea4c13fd70 2023-09-30T00:19:17.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.032+0000 7fea40ff9700 1 -- 172.21.15.94:0/1679375635 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea44012b70 con 0x7fea4c13fd70 2023-09-30T00:19:17.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.032+0000 7fea530a3700 1 -- 172.21.15.94:0/1679375635 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4c0a62a0 con 0x7fea4c13fd70 2023-09-30T00:19:17.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.033+0000 7fea530a3700 1 -- 172.21.15.94:0/1679375635 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4c071630 con 0x7fea4c13fd70 2023-09-30T00:19:17.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.034+0000 7fea40ff9700 1 -- 172.21.15.94:0/1679375635 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fea44003030 con 0x7fea4c13fd70 2023-09-30T00:19:17.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.034+0000 7fea40ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:17.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.034+0000 7fea40ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:17.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.034+0000 7fea40ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:17.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.034+0000 7fea40ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:17.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.034+0000 7fea40ff9700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fea38098830 0x7fea3809ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:17.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.034+0000 7fea50e3f700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fea38098830 0x7fea3809ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:17.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.034+0000 7fea40ff9700 1 -- 172.21.15.94:0/1679375635 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fea440c0500 con 0x7fea4c13fd70 2023-09-30T00:19:17.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.035+0000 7fea50e3f700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fea38098830 0x7fea3809ace0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fea480099e0 tx=0x7fea48017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:17.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.039+0000 7fea40ff9700 1 -- 172.21.15.94:0/1679375635 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea4408daa0 con 0x7fea4c13fd70 2023-09-30T00:19:17.263 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.262+0000 7fea530a3700 1 -- 172.21.15.94:0/1679375635 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fea4c146a00 con 0x7fea4c13fd70 2023-09-30T00:19:17.263 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.263+0000 7fea40ff9700 1 -- 172.21.15.94:0/1679375635 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fea440910c0 con 0x7fea4c13fd70 2023-09-30T00:19:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:17.264 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:17.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 10 mgrc shutdown 2023-09-30T00:19:17.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 1 -- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fea38098830 msgr2=0x7fea3809ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:17.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fea38098830 0x7fea3809ace0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fea480099e0 tx=0x7fea48017000 comp rx=0 tx=0).stop 2023-09-30T00:19:17.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 1 -- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c13fd70 msgr2=0x7fea4c147df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:17.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c13fd70 0x7fea4c147df0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fea4400c840 tx=0x7fea44001670 comp rx=0 tx=0).stop 2023-09-30T00:19:17.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 1 -- 172.21.15.94:0/1679375635 shutdown_connections 2023-09-30T00:19:17.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fea38098830 0x7fea3809ace0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:17.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fea4c13fd70 0x7fea4c147df0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:17.266 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea4c070c50 0x7fea4c146cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:17.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 1 --2- 172.21.15.94:0/1679375635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea4c0b7160 0x7fea4c140a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:17.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 1 -- 172.21.15.94:0/1679375635 >> 172.21.15.94:0/1679375635 conn(0x7fea4c06a090 msgr2=0x7fea4c13d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:17.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 1 -- 172.21.15.94:0/1679375635 shutdown_connections 2023-09-30T00:19:17.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:17.265+0000 7fea530a3700 1 -- 172.21.15.94:0/1679375635 wait complete. 2023-09-30T00:19:20.278 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.404+0000 7ff8cefb6700 1 Processor -- start 2023-09-30T00:19:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.404+0000 7ff8cefb6700 1 -- start start 2023-09-30T00:19:20.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.404+0000 7ff8cefb6700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff8c80b7160 0x7ff8c80a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:20.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.404+0000 7ff8cefb6700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff8c813fd70 0x7ff8c8140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:20.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.404+0000 7ff8cefb6700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c80a1310 con 0x7ff8c8070c50 2023-09-30T00:19:20.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.404+0000 7ff8cefb6700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8c8071020 con 0x7ff8c80b7160 2023-09-30T00:19:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.404+0000 7ff8cefb6700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8c80b76a0 con 0x7ff8c813fd70 2023-09-30T00:19:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8ccd52700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff8c8070c50 0x7ff8c80a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38676/0 (socket says 172.21.15.94:38676) 2023-09-30T00:19:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8ccd52700 1 -- 172.21.15.94:0/1633729099 learned_addr learned my addr 172.21.15.94:0/1633729099 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:19:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8c7fff700 1 --2- 172.21.15.94:0/1633729099 >> v2:172.21.15.94:3300/0 conn(0x7ff8c80b7160 0x7ff8c80a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8cd553700 1 --2- 172.21.15.94:0/1633729099 >> v2:172.21.15.121:3300/0 conn(0x7ff8c813fd70 0x7ff8c8140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8c77fe700 1 -- 172.21.15.94:0/1633729099 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629826440 0 0) 0x7ff8bc001110 con 0x7ff8c8070c50 2023-09-30T00:19:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8c77fe700 1 -- 172.21.15.94:0/1633729099 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b8003570 con 0x7ff8c8070c50 2023-09-30T00:19:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8c7fff700 1 -- 172.21.15.94:0/1633729099 >> v2:172.21.15.121:3300/0 conn(0x7ff8c813fd70 msgr2=0x7ff8c8140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8c7fff700 1 --2- 172.21.15.94:0/1633729099 >> v2:172.21.15.121:3300/0 conn(0x7ff8c813fd70 0x7ff8c8140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8c7fff700 1 -- 172.21.15.94:0/1633729099 >> v1:172.21.15.94:6789/0 conn(0x7ff8c8070c50 legacy=0x7ff8c80a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8c7fff700 1 -- 172.21.15.94:0/1633729099 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c80b7940 con 0x7ff8c80b7160 2023-09-30T00:19:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8c7fff700 1 --2- 172.21.15.94:0/1633729099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8c80b7160 0x7ff8c80a0dd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80a31a0 tx=0x7ff8b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c3b354a359d3cb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8c77fe700 1 -- 172.21.15.94:0/1633729099 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b400b690 con 0x7ff8c80b7160 2023-09-30T00:19:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8c77fe700 1 -- 172.21.15.94:0/1633729099 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b400b7f0 con 0x7ff8c80b7160 2023-09-30T00:19:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.405+0000 7ff8c77fe700 1 -- 172.21.15.94:0/1633729099 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b400bb90 con 0x7ff8c80b7160 2023-09-30T00:19:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.406+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1633729099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8c80b7160 msgr2=0x7ff8c80a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.406+0000 7ff8cefb6700 1 --2- 172.21.15.94:0/1633729099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8c80b7160 0x7ff8c80a0dd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80a31a0 tx=0x7ff8b40091a0 comp rx=0 tx=0).stop 2023-09-30T00:19:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.406+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1633729099 shutdown_connections 2023-09-30T00:19:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.406+0000 7ff8cefb6700 1 --2- 172.21.15.94:0/1633729099 >> v2:172.21.15.121:3300/0 conn(0x7ff8c813fd70 0x7ff8c8140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.406+0000 7ff8cefb6700 1 --2- 172.21.15.94:0/1633729099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8c80b7160 0x7ff8c80a0dd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.406+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1633729099 >> 172.21.15.94:0/1633729099 conn(0x7ff8c806a090 msgr2=0x7ff8c80702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.406+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1633729099 shutdown_connections 2023-09-30T00:19:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.406+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1633729099 wait complete. 2023-09-30T00:19:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.407+0000 7ff8cefb6700 1 Processor -- start 2023-09-30T00:19:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.407+0000 7ff8cefb6700 1 -- start start 2023-09-30T00:19:20.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.407+0000 7ff8cefb6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8c8070c50 0x7ff8c8143590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:20.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.407+0000 7ff8cefb6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8c80b7160 0x7ff8c809bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:20.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.407+0000 7ff8cefb6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8c813fd70 0x7ff8c8139e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:20.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.407+0000 7ff8cefb6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c80a3260 con 0x7ff8c8070c50 2023-09-30T00:19:20.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.407+0000 7ff8cefb6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c8143dc0 con 0x7ff8c80b7160 2023-09-30T00:19:20.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.407+0000 7ff8cefb6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c8143ad0 con 0x7ff8c813fd70 2023-09-30T00:19:20.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.407+0000 7ff8ccd52700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8c8070c50 0x7ff8c8143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:20.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8ccd52700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8c8070c50 0x7ff8c8143590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50248/0 (socket says 172.21.15.94:50248) 2023-09-30T00:19:20.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8ccd52700 1 -- 172.21.15.94:0/1397629242 learned_addr learned my addr 172.21.15.94:0/1397629242 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:20.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8cd553700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8c813fd70 0x7ff8c8139e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:20.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8c7fff700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8c80b7160 0x7ff8c809bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:20.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8ccd52700 1 -- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8c813fd70 msgr2=0x7ff8c8139e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:20.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8ccd52700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8c813fd70 0x7ff8c8139e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:20.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8ccd52700 1 -- 172.21.15.94:0/1397629242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8c80b7160 msgr2=0x7ff8c809bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:20.413 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8ccd52700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8c80b7160 0x7ff8c809bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:20.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8ccd52700 1 -- 172.21.15.94:0/1397629242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b4009420 con 0x7ff8c8070c50 2023-09-30T00:19:20.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8ccd52700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8c8070c50 0x7ff8c8143590 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc00c7c0 tx=0x7ff8bc007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:20.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8c4ff9700 1 -- 172.21.15.94:0/1397629242 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc005c00 con 0x7ff8c8070c50 2023-09-30T00:19:20.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8c4ff9700 1 -- 172.21.15.94:0/1397629242 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8bc005d60 con 0x7ff8c8070c50 2023-09-30T00:19:20.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1397629242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c809c170 con 0x7ff8c8070c50 2023-09-30T00:19:20.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8c4ff9700 1 -- 172.21.15.94:0/1397629242 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc007320 con 0x7ff8c8070c50 2023-09-30T00:19:20.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.408+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1397629242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c80a6200 con 0x7ff8c8070c50 2023-09-30T00:19:20.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.409+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1397629242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c8071630 con 0x7ff8c8070c50 2023-09-30T00:19:20.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.410+0000 7ff8c4ff9700 1 -- 172.21.15.94:0/1397629242 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ff8bc01b3b0 con 0x7ff8c8070c50 2023-09-30T00:19:20.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.410+0000 7ff8c4ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:20.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.411+0000 7ff8c4ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:20.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.411+0000 7ff8c4ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:20.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.411+0000 7ff8c4ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:20.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.411+0000 7ff8c4ff9700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff8ac098780 0x7ff8ac09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:20.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.411+0000 7ff8c7fff700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff8ac098780 0x7ff8ac09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:20.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.411+0000 7ff8c4ff9700 1 -- 172.21.15.94:0/1397629242 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7ff8bc0bfc70 con 0x7ff8c8070c50 2023-09-30T00:19:20.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.411+0000 7ff8c7fff700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff8ac098780 0x7ff8ac09ac30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40093f0 tx=0x7ff8b4017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:20.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.414+0000 7ff8c4ff9700 1 -- 172.21.15.94:0/1397629242 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8bc08d340 con 0x7ff8c8070c50 2023-09-30T00:19:20.641 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.641+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1397629242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff8c80a1310 con 0x7ff8c8070c50 2023-09-30T00:19:20.642 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.641+0000 7ff8c4ff9700 1 -- 172.21.15.94:0/1397629242 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff8bc017100 con 0x7ff8c8070c50 2023-09-30T00:19:20.642 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:20.642 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:20.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 10 mgrc shutdown 2023-09-30T00:19:20.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff8ac098780 msgr2=0x7ff8ac09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:20.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff8ac098780 0x7ff8ac09ac30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40093f0 tx=0x7ff8b4017000 comp rx=0 tx=0).stop 2023-09-30T00:19:20.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8c8070c50 msgr2=0x7ff8c8143590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:20.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8c8070c50 0x7ff8c8143590 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc00c7c0 tx=0x7ff8bc007c70 comp rx=0 tx=0).stop 2023-09-30T00:19:20.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1397629242 shutdown_connections 2023-09-30T00:19:20.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff8ac098780 0x7ff8ac09ac30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:20.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8c813fd70 0x7ff8c8139e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:20.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8c8070c50 0x7ff8c8143590 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:20.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 1 --2- 172.21.15.94:0/1397629242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8c80b7160 0x7ff8c809bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:20.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1397629242 >> 172.21.15.94:0/1397629242 conn(0x7ff8c806a090 msgr2=0x7ff8c8067cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:20.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1397629242 shutdown_connections 2023-09-30T00:19:20.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:20.643+0000 7ff8cefb6700 1 -- 172.21.15.94:0/1397629242 wait complete. 2023-09-30T00:19:23.656 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:23.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.783+0000 7fd6c9836700 1 Processor -- start 2023-09-30T00:19:23.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.783+0000 7fd6c9836700 1 -- start start 2023-09-30T00:19:23.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6c9836700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd6c409de30 0x7fd6c4140b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:23.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6c9836700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c406eab0 0x7fd6c409f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:23.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6c9836700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c409fdc0 con 0x7fd6c40a2ef0 2023-09-30T00:19:23.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6c9836700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c4141180 con 0x7fd6c409de30 2023-09-30T00:19:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6c9836700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c4141440 con 0x7fd6c406eab0 2023-09-30T00:19:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6ba7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c406eab0 0x7fd6c409f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6ba7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c406eab0 0x7fd6c409f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44664/0 (socket says 172.21.15.94:44664) 2023-09-30T00:19:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6ba7fc700 1 -- 172.21.15.94:0/3349741388 learned_addr learned my addr 172.21.15.94:0/3349741388 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6c2ffd700 1 --2- 172.21.15.94:0/3349741388 >> v2:172.21.15.121:3300/0 conn(0x7fd6c409de30 0x7fd6c4140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6ba7fc700 1 -- 172.21.15.94:0/3349741388 >> v2:172.21.15.121:3300/0 conn(0x7fd6c409de30 msgr2=0x7fd6c4140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6ba7fc700 1 --2- 172.21.15.94:0/3349741388 >> v2:172.21.15.121:3300/0 conn(0x7fd6c409de30 0x7fd6c4140b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6ba7fc700 1 -- 172.21.15.94:0/3349741388 >> v1:172.21.15.121:6789/0 conn(0x7fd6c40a2ef0 legacy=0x7fd6c40a4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6ba7fc700 1 -- 172.21.15.94:0/3349741388 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c4141700 con 0x7fd6c406eab0 2023-09-30T00:19:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6ba7fc700 1 --2- 172.21.15.94:0/3349741388 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c406eab0 0x7fd6c409f880 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0099b0 tx=0x7fd6ac009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5504da0c0f33ed6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:23.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6c27fc700 1 -- 172.21.15.94:0/3349741388 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac00b8b0 con 0x7fd6c406eab0 2023-09-30T00:19:23.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6c27fc700 1 -- 172.21.15.94:0/3349741388 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ac00ba10 con 0x7fd6c406eab0 2023-09-30T00:19:23.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.784+0000 7fd6c27fc700 1 -- 172.21.15.94:0/3349741388 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac00bdb0 con 0x7fd6c406eab0 2023-09-30T00:19:23.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.785+0000 7fd6c9836700 1 -- 172.21.15.94:0/3349741388 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c406eab0 msgr2=0x7fd6c409f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:23.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.785+0000 7fd6c9836700 1 --2- 172.21.15.94:0/3349741388 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c406eab0 0x7fd6c409f880 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0099b0 tx=0x7fd6ac009250 comp rx=0 tx=0).stop 2023-09-30T00:19:23.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.785+0000 7fd6c9836700 1 -- 172.21.15.94:0/3349741388 shutdown_connections 2023-09-30T00:19:23.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.785+0000 7fd6c9836700 1 --2- 172.21.15.94:0/3349741388 >> v2:172.21.15.121:3300/0 conn(0x7fd6c409de30 0x7fd6c4140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:23.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.785+0000 7fd6c9836700 1 --2- 172.21.15.94:0/3349741388 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c406eab0 0x7fd6c409f880 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:23.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.785+0000 7fd6c9836700 1 -- 172.21.15.94:0/3349741388 >> 172.21.15.94:0/3349741388 conn(0x7fd6c40b3520 msgr2=0x7fd6c4067620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:23.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.785+0000 7fd6c9836700 1 -- 172.21.15.94:0/3349741388 shutdown_connections 2023-09-30T00:19:23.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.785+0000 7fd6c9836700 1 -- 172.21.15.94:0/3349741388 wait complete. 2023-09-30T00:19:23.791 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.786+0000 7fd6c9836700 1 Processor -- start 2023-09-30T00:19:23.791 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.786+0000 7fd6c9836700 1 -- start start 2023-09-30T00:19:23.791 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6c9836700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6c406eab0 0x7fd6c40a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:23.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6c9836700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c409de30 0x7fd6c40987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:23.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6c9836700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6c40a2ef0 0x7fd6c4096c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:23.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6c9836700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c413c4a0 con 0x7fd6c40a2ef0 2023-09-30T00:19:23.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6c9836700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c4097fd0 con 0x7fd6c406eab0 2023-09-30T00:19:23.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6c9836700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c414d7e0 con 0x7fd6c409de30 2023-09-30T00:19:23.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6ba7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c409de30 0x7fd6c40987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:23.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6c37fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6c40a2ef0 0x7fd6c4096c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:23.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6ba7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c409de30 0x7fd6c40987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44666/0 (socket says 172.21.15.94:44666) 2023-09-30T00:19:23.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6ba7fc700 1 -- 172.21.15.94:0/1870939151 learned_addr learned my addr 172.21.15.94:0/1870939151 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:23.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6c2ffd700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6c406eab0 0x7fd6c40a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:23.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6ba7fc700 1 -- 172.21.15.94:0/1870939151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6c406eab0 msgr2=0x7fd6c40a1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:23.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6ba7fc700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6c406eab0 0x7fd6c40a1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:23.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6ba7fc700 1 -- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6c40a2ef0 msgr2=0x7fd6c4096c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:23.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6ba7fc700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6c40a2ef0 0x7fd6c4096c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:23.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6ba7fc700 1 -- 172.21.15.94:0/1870939151 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac0094f0 con 0x7fd6c409de30 2023-09-30T00:19:23.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6c37fe700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6c40a2ef0 0x7fd6c4096c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:23.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6c2ffd700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6c406eab0 0x7fd6c40a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:19:23.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6ba7fc700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c409de30 0x7fd6c40987d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac009290 tx=0x7fd6ac016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:23.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.787+0000 7fd6bb7fe700 1 -- 172.21.15.94:0/1870939151 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac020070 con 0x7fd6c409de30 2023-09-30T00:19:23.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.788+0000 7fd6bb7fe700 1 -- 172.21.15.94:0/1870939151 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ac011e90 con 0x7fd6c409de30 2023-09-30T00:19:23.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.788+0000 7fd6c9836700 1 -- 172.21.15.94:0/1870939151 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c414daa0 con 0x7fd6c409de30 2023-09-30T00:19:23.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.788+0000 7fd6bb7fe700 1 -- 172.21.15.94:0/1870939151 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac01a7b0 con 0x7fd6c409de30 2023-09-30T00:19:23.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.788+0000 7fd6c9836700 1 -- 172.21.15.94:0/1870939151 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c4098d10 con 0x7fd6c409de30 2023-09-30T00:19:23.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.789+0000 7fd6c9836700 1 -- 172.21.15.94:0/1870939151 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c4071630 con 0x7fd6c409de30 2023-09-30T00:19:23.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.789+0000 7fd6bb7fe700 1 -- 172.21.15.94:0/1870939151 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fd6ac01aba0 con 0x7fd6c409de30 2023-09-30T00:19:23.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.789+0000 7fd6bb7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:23.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.790+0000 7fd6bb7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:23.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.790+0000 7fd6bb7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:23.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.790+0000 7fd6bb7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:23.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.790+0000 7fd6bb7fe700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd6a4098780 0x7fd6a409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:23.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.790+0000 7fd6c2ffd700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd6a4098780 0x7fd6a409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:23.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.790+0000 7fd6bb7fe700 1 -- 172.21.15.94:0/1870939151 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fd6ac0c3960 con 0x7fd6c409de30 2023-09-30T00:19:23.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.790+0000 7fd6c2ffd700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd6a4098780 0x7fd6a409ac30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4006fd0 tx=0x7fd6b4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:23.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:23.794+0000 7fd6bb7fe700 1 -- 172.21.15.94:0/1870939151 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd6ac091030 con 0x7fd6c409de30 2023-09-30T00:19:24.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.015+0000 7fd6c9836700 1 -- 172.21.15.94:0/1870939151 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd6c409e200 con 0x7fd6c409de30 2023-09-30T00:19:24.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.016+0000 7fd6bb7fe700 1 -- 172.21.15.94:0/1870939151 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd6ac02a080 con 0x7fd6c409de30 2023-09-30T00:19:24.016 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:24.016 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:24.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 10 mgrc shutdown 2023-09-30T00:19:24.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 1 -- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd6a4098780 msgr2=0x7fd6a409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:24.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd6a4098780 0x7fd6a409ac30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4006fd0 tx=0x7fd6b4008040 comp rx=0 tx=0).stop 2023-09-30T00:19:24.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 1 -- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c409de30 msgr2=0x7fd6c40987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:24.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c409de30 0x7fd6c40987d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac009290 tx=0x7fd6ac016000 comp rx=0 tx=0).stop 2023-09-30T00:19:24.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 1 -- 172.21.15.94:0/1870939151 shutdown_connections 2023-09-30T00:19:24.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd6a4098780 0x7fd6a409ac30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:24.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6c40a2ef0 0x7fd6c4096c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:24.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6c406eab0 0x7fd6c40a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:24.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 1 --2- 172.21.15.94:0/1870939151 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd6c409de30 0x7fd6c40987d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 1 -- 172.21.15.94:0/1870939151 >> 172.21.15.94:0/1870939151 conn(0x7fd6c40b3520 msgr2=0x7fd6c40b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 1 -- 172.21.15.94:0/1870939151 shutdown_connections 2023-09-30T00:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:24.018+0000 7fd6c9836700 1 -- 172.21.15.94:0/1870939151 wait complete. 2023-09-30T00:19:27.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:27.165 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.157+0000 7f740542a700 1 Processor -- start 2023-09-30T00:19:27.165 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.157+0000 7f740542a700 1 -- start start 2023-09-30T00:19:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f740542a700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f74000b6e50 0x7f74000a1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f740542a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000a2140 0x7f74000a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f740542a700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f740013fe20 0x7f7400140260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f740542a700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74001407a0 con 0x7f740013fe20 2023-09-30T00:19:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f740542a700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74000b7220 con 0x7f74000b6e50 2023-09-30T00:19:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f740542a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74000a2870 con 0x7f74000a2140 2023-09-30T00:19:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f73ff7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000a2140 0x7f74000a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f7404c29700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f740013fe20 0x7f7400140260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f73ff7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000a2140 0x7f74000a2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:35102/0 (socket says 172.21.15.94:35102) 2023-09-30T00:19:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f73ff7fe700 1 -- 172.21.15.94:0/1654635990 learned_addr learned my addr 172.21.15.94:0/1654635990 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f73fffff700 1 --2- 172.21.15.94:0/1654635990 >> v2:172.21.15.121:3300/0 conn(0x7f74000b6e50 0x7f74000a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f73ff7fe700 1 -- 172.21.15.94:0/1654635990 >> v2:172.21.15.121:3300/0 conn(0x7f74000b6e50 msgr2=0x7f74000a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:27.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f73ff7fe700 1 --2- 172.21.15.94:0/1654635990 >> v2:172.21.15.121:3300/0 conn(0x7f74000b6e50 0x7f74000a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:27.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f73ff7fe700 1 -- 172.21.15.94:0/1654635990 >> v2:172.21.15.94:3300/0 conn(0x7f740013fe20 msgr2=0x7f7400140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:27.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f73ff7fe700 1 --2- 172.21.15.94:0/1654635990 >> v2:172.21.15.94:3300/0 conn(0x7f740013fe20 0x7f7400140260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:27.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f73ff7fe700 1 -- 172.21.15.94:0/1654635990 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74000a2b30 con 0x7f74000a2140 2023-09-30T00:19:27.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f7404c29700 1 --2- 172.21.15.94:0/1654635990 >> v2:172.21.15.94:3300/0 conn(0x7f740013fe20 0x7f7400140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:27.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.158+0000 7f73ff7fe700 1 --2- 172.21.15.94:0/1654635990 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000a2140 0x7f74000a2fb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7400071250 tx=0x7f73f4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b942d600c0dfe29 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:27.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.159+0000 7f73feffd700 1 -- 172.21.15.94:0/1654635990 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4009e70 con 0x7f74000a2140 2023-09-30T00:19:27.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.159+0000 7f73feffd700 1 -- 172.21.15.94:0/1654635990 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f400b860 con 0x7f74000a2140 2023-09-30T00:19:27.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.159+0000 7f73feffd700 1 -- 172.21.15.94:0/1654635990 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f400bc00 con 0x7f74000a2140 2023-09-30T00:19:27.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.159+0000 7f740542a700 1 -- 172.21.15.94:0/1654635990 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000a2140 msgr2=0x7f74000a2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:27.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.159+0000 7f740542a700 1 --2- 172.21.15.94:0/1654635990 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000a2140 0x7f74000a2fb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7400071250 tx=0x7f73f4009250 comp rx=0 tx=0).stop 2023-09-30T00:19:27.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.159+0000 7f740542a700 1 -- 172.21.15.94:0/1654635990 shutdown_connections 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.159+0000 7f740542a700 1 --2- 172.21.15.94:0/1654635990 >> v2:172.21.15.94:3300/0 conn(0x7f740013fe20 0x7f7400140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.159+0000 7f740542a700 1 --2- 172.21.15.94:0/1654635990 >> v2:172.21.15.121:3300/0 conn(0x7f74000b6e50 0x7f74000a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.159+0000 7f740542a700 1 --2- 172.21.15.94:0/1654635990 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000a2140 0x7f74000a2fb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.159+0000 7f740542a700 1 -- 172.21.15.94:0/1654635990 >> 172.21.15.94:0/1654635990 conn(0x7f74000702f0 msgr2=0x7f74000706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.159+0000 7f740542a700 1 -- 172.21.15.94:0/1654635990 shutdown_connections 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.159+0000 7f740542a700 1 -- 172.21.15.94:0/1654635990 wait complete. 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.160+0000 7f740542a700 1 Processor -- start 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.160+0000 7f740542a700 1 -- start start 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f740542a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74000a2140 0x7f7400142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f740542a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000b6e50 0x7f74000a3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f740542a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f740013fe20 0x7f740013ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f740542a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7400142260 con 0x7f740013fe20 2023-09-30T00:19:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f740542a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74000a3fa0 con 0x7f74000a2140 2023-09-30T00:19:27.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f740542a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74000a41b0 con 0x7f74000b6e50 2023-09-30T00:19:27.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f7404c29700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f740013fe20 0x7f740013ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:27.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f73ff7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000b6e50 0x7f74000a3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:27.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f7404c29700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f740013fe20 0x7f740013ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51114/0 (socket says 172.21.15.94:51114) 2023-09-30T00:19:27.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f7404c29700 1 -- 172.21.15.94:0/2401845008 learned_addr learned my addr 172.21.15.94:0/2401845008 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:27.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f73ff7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000b6e50 0x7f74000a3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:35110/0 (socket says 172.21.15.94:35110) 2023-09-30T00:19:27.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f73fffff700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74000a2140 0x7f7400142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:27.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f73ff7fe700 1 -- 172.21.15.94:0/2401845008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74000a2140 msgr2=0x7f7400142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:27.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f73ff7fe700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74000a2140 0x7f7400142fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:27.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f73ff7fe700 1 -- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f740013fe20 msgr2=0x7f740013ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:27.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f73ff7fe700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f740013fe20 0x7f740013ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:27.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f73ff7fe700 1 -- 172.21.15.94:0/2401845008 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f8009a20 con 0x7f74000b6e50 2023-09-30T00:19:27.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f7404c29700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f740013fe20 0x7f740013ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:27.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f73ff7fe700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000b6e50 0x7f74000a3a60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f74000a0d50 tx=0x7f73f4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:27.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.161+0000 7f73e7fff700 1 -- 172.21.15.94:0/2401845008 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f400bbe0 con 0x7f74000b6e50 2023-09-30T00:19:27.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.162+0000 7f73e7fff700 1 -- 172.21.15.94:0/2401845008 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f4000c90 con 0x7f74000b6e50 2023-09-30T00:19:27.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.162+0000 7f740542a700 1 -- 172.21.15.94:0/2401845008 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f40094f0 con 0x7f74000b6e50 2023-09-30T00:19:27.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.162+0000 7f73e7fff700 1 -- 172.21.15.94:0/2401845008 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4011650 con 0x7f74000b6e50 2023-09-30T00:19:27.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.162+0000 7f740542a700 1 -- 172.21.15.94:0/2401845008 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f740013f780 con 0x7f74000b6e50 2023-09-30T00:19:27.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.163+0000 7f740542a700 1 -- 172.21.15.94:0/2401845008 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7400066e50 con 0x7f74000b6e50 2023-09-30T00:19:27.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.163+0000 7f73e7fff700 1 -- 172.21.15.94:0/2401845008 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f73f4011a40 con 0x7f74000b6e50 2023-09-30T00:19:27.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.163+0000 7f73e7fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:27.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.164+0000 7f73e7fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:27.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.164+0000 7f73e7fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:27.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.164+0000 7f73e7fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:27.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.164+0000 7f73e7fff700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f73ec098780 0x7f73ec09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:27.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.164+0000 7f73fffff700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f73ec098780 0x7f73ec09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:27.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.164+0000 7f73e7fff700 1 -- 172.21.15.94:0/2401845008 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f73f40c1810 con 0x7f74000b6e50 2023-09-30T00:19:27.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.164+0000 7f73fffff700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f73ec098780 0x7f73ec09ac30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f73f0006fd0 tx=0x7f73f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:27.171 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.168+0000 7f73e7fff700 1 -- 172.21.15.94:0/2401845008 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73f408eee0 con 0x7f74000b6e50 2023-09-30T00:19:27.392 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.390+0000 7f740542a700 1 -- 172.21.15.94:0/2401845008 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f74000871f0 con 0x7f74000b6e50 2023-09-30T00:19:27.392 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.391+0000 7f73e7fff700 1 -- 172.21.15.94:0/2401845008 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f73f4092500 con 0x7f74000b6e50 2023-09-30T00:19:27.393 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:27.393 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:27.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 10 mgrc shutdown 2023-09-30T00:19:27.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 1 -- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f73ec098780 msgr2=0x7f73ec09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:27.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f73ec098780 0x7f73ec09ac30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f73f0006fd0 tx=0x7f73f0008040 comp rx=0 tx=0).stop 2023-09-30T00:19:27.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 1 -- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000b6e50 msgr2=0x7f74000a3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:27.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000b6e50 0x7f74000a3a60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f74000a0d50 tx=0x7f73f4017000 comp rx=0 tx=0).stop 2023-09-30T00:19:27.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 1 -- 172.21.15.94:0/2401845008 shutdown_connections 2023-09-30T00:19:27.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f73ec098780 0x7f73ec09ac30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:27.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f740013fe20 0x7f740013ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:27.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74000a2140 0x7f7400142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:27.396 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 1 --2- 172.21.15.94:0/2401845008 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f74000b6e50 0x7f74000a3a60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:27.396 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 1 -- 172.21.15.94:0/2401845008 >> 172.21.15.94:0/2401845008 conn(0x7f74000702f0 msgr2=0x7f74000630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:27.396 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 1 -- 172.21.15.94:0/2401845008 shutdown_connections 2023-09-30T00:19:27.396 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:27.393+0000 7f740542a700 1 -- 172.21.15.94:0/2401845008 wait complete. 2023-09-30T00:19:30.406 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:30.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.530+0000 7f217121b700 1 Processor -- start 2023-09-30T00:19:30.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.530+0000 7f217121b700 1 -- start start 2023-09-30T00:19:30.535 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.531+0000 7f217121b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 0x7f216c0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:30.535 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.531+0000 7f217121b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f216c0b7160 0x7f216c0a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:30.535 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.531+0000 7f217121b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f216c13fd70 0x7f216c1401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:30.535 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.531+0000 7f217121b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f216c0a1310 con 0x7f216c13fd70 2023-09-30T00:19:30.535 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.531+0000 7f217121b700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f216c071020 con 0x7f216c0b7160 2023-09-30T00:19:30.535 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.531+0000 7f217121b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0b76a0 con 0x7f216c070c50 2023-09-30T00:19:30.535 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.531+0000 7f216ad9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 0x7f216c0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:30.535 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.531+0000 7f216b59e700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f216c13fd70 0x7f216c1401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:30.535 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.531+0000 7f216ad9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 0x7f216c0a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:35120/0 (socket says 172.21.15.94:35120) 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.531+0000 7f216ad9d700 1 -- 172.21.15.94:0/653175978 learned_addr learned my addr 172.21.15.94:0/653175978 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.531+0000 7f216259c700 1 --2- 172.21.15.94:0/653175978 >> v2:172.21.15.121:3300/0 conn(0x7f216c0b7160 0x7f216c0a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f216b59e700 1 -- 172.21.15.94:0/653175978 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 msgr2=0x7f216c0a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f216b59e700 1 --2- 172.21.15.94:0/653175978 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 0x7f216c0a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f216b59e700 1 -- 172.21.15.94:0/653175978 >> v2:172.21.15.121:3300/0 conn(0x7f216c0b7160 msgr2=0x7f216c0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f216b59e700 1 --2- 172.21.15.94:0/653175978 >> v2:172.21.15.121:3300/0 conn(0x7f216c0b7160 0x7f216c0a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f216b59e700 1 -- 172.21.15.94:0/653175978 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c0b7940 con 0x7f216c13fd70 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f216ad9d700 1 --2- 172.21.15.94:0/653175978 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 0x7f216c0a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f216b59e700 1 --2- 172.21.15.94:0/653175978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c13fd70 0x7f216c1401d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f216c064660 tx=0x7f2164009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c206043629e8311e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f216a59c700 1 -- 172.21.15.94:0/653175978 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216400e070 con 0x7f216c13fd70 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f216a59c700 1 -- 172.21.15.94:0/653175978 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216400bb40 con 0x7f216c13fd70 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f216a59c700 1 -- 172.21.15.94:0/653175978 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2164012430 con 0x7f216c13fd70 2023-09-30T00:19:30.536 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f217121b700 1 -- 172.21.15.94:0/653175978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c13fd70 msgr2=0x7f216c1401d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:30.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f217121b700 1 --2- 172.21.15.94:0/653175978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c13fd70 0x7f216c1401d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f216c064660 tx=0x7f2164009780 comp rx=0 tx=0).stop 2023-09-30T00:19:30.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f217121b700 1 -- 172.21.15.94:0/653175978 shutdown_connections 2023-09-30T00:19:30.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f217121b700 1 --2- 172.21.15.94:0/653175978 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c13fd70 0x7f216c1401d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:30.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f217121b700 1 --2- 172.21.15.94:0/653175978 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 0x7f216c0a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:30.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f217121b700 1 --2- 172.21.15.94:0/653175978 >> v2:172.21.15.121:3300/0 conn(0x7f216c0b7160 0x7f216c0a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:30.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f217121b700 1 -- 172.21.15.94:0/653175978 >> 172.21.15.94:0/653175978 conn(0x7f216c06a090 msgr2=0x7f216c0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:30.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f217121b700 1 -- 172.21.15.94:0/653175978 shutdown_connections 2023-09-30T00:19:30.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.532+0000 7f217121b700 1 -- 172.21.15.94:0/653175978 wait complete. 2023-09-30T00:19:30.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f217121b700 1 Processor -- start 2023-09-30T00:19:30.537 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f217121b700 1 -- start start 2023-09-30T00:19:30.539 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f217121b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 0x7f216c0a8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:30.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f217121b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f216c0b7160 0x7f216c153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:30.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f217121b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c13fd70 0x7f216c14f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:30.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f217121b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0a58d0 con 0x7f216c13fd70 2023-09-30T00:19:30.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f217121b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0a6530 con 0x7f216c0b7160 2023-09-30T00:19:30.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f217121b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0a6a00 con 0x7f216c070c50 2023-09-30T00:19:30.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f216ad9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 0x7f216c0a8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:30.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f216b59e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c13fd70 0x7f216c14f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:30.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f216ad9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 0x7f216c0a8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:35132/0 (socket says 172.21.15.94:35132) 2023-09-30T00:19:30.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f216ad9d700 1 -- 172.21.15.94:0/1039117415 learned_addr learned my addr 172.21.15.94:0/1039117415 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:30.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.534+0000 7f216259c700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f216c0b7160 0x7f216c153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:30.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.535+0000 7f216ad9d700 1 -- 172.21.15.94:0/1039117415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f216c0b7160 msgr2=0x7f216c153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:30.540 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.535+0000 7f216ad9d700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f216c0b7160 0x7f216c153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:30.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.535+0000 7f216ad9d700 1 -- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c13fd70 msgr2=0x7f216c14f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:30.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.535+0000 7f216ad9d700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c13fd70 0x7f216c14f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:30.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.535+0000 7f216ad9d700 1 -- 172.21.15.94:0/1039117415 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c0094f0 con 0x7f216c070c50 2023-09-30T00:19:30.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.535+0000 7f216b59e700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c13fd70 0x7f216c14f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:30.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.535+0000 7f216ad9d700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 0x7f216c0a8930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f215c00ec80 tx=0x7f215c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:30.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.535+0000 7f21637fe700 1 -- 172.21.15.94:0/1039117415 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c00c8e0 con 0x7f216c070c50 2023-09-30T00:19:30.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.535+0000 7f21637fe700 1 -- 172.21.15.94:0/1039117415 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215c00ca40 con 0x7f216c070c50 2023-09-30T00:19:30.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.535+0000 7f21637fe700 1 -- 172.21.15.94:0/1039117415 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c00cde0 con 0x7f216c070c50 2023-09-30T00:19:30.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.535+0000 7f217121b700 1 -- 172.21.15.94:0/1039117415 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2164009a20 con 0x7f216c070c50 2023-09-30T00:19:30.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.535+0000 7f217121b700 1 -- 172.21.15.94:0/1039117415 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216c142b20 con 0x7f216c070c50 2023-09-30T00:19:30.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.536+0000 7f217121b700 1 -- 172.21.15.94:0/1039117415 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216c142d40 con 0x7f216c070c50 2023-09-30T00:19:30.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.537+0000 7f21637fe700 1 -- 172.21.15.94:0/1039117415 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f215c013680 con 0x7f216c070c50 2023-09-30T00:19:30.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.537+0000 7f21637fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:30.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.537+0000 7f21637fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:30.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.537+0000 7f21637fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:30.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.537+0000 7f21637fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:30.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.537+0000 7f21637fe700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f214c0987b0 0x7f214c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:30.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.537+0000 7f216259c700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f214c0987b0 0x7f214c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:30.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.537+0000 7f21637fe700 1 -- 172.21.15.94:0/1039117415 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f215c0c0a40 con 0x7f216c070c50 2023-09-30T00:19:30.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.538+0000 7f216259c700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f214c0987b0 0x7f214c09ac60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f216c068ea0 tx=0x7f21540091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:30.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.540+0000 7f21637fe700 1 -- 172.21.15.94:0/1039117415 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f215c08e080 con 0x7f216c070c50 2023-09-30T00:19:30.766 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.765+0000 7f217121b700 1 -- 172.21.15.94:0/1039117415 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f216c142d40 con 0x7f216c070c50 2023-09-30T00:19:30.766 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.765+0000 7f21637fe700 1 -- 172.21.15.94:0/1039117415 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f215c017100 con 0x7f216c070c50 2023-09-30T00:19:30.767 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:30.767 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:30.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 10 mgrc shutdown 2023-09-30T00:19:30.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 1 -- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f214c0987b0 msgr2=0x7f214c09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:30.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f214c0987b0 0x7f214c09ac60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f216c068ea0 tx=0x7f21540091a0 comp rx=0 tx=0).stop 2023-09-30T00:19:30.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 1 -- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 msgr2=0x7f216c0a8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:30.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 0x7f216c0a8930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f215c00ec80 tx=0x7f215c009250 comp rx=0 tx=0).stop 2023-09-30T00:19:30.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 1 -- 172.21.15.94:0/1039117415 shutdown_connections 2023-09-30T00:19:30.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f214c0987b0 0x7f214c09ac60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:30.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f216c070c50 0x7f216c0a8930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:30.769 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f216c13fd70 0x7f216c14f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 1 --2- 172.21.15.94:0/1039117415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f216c0b7160 0x7f216c153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 1 -- 172.21.15.94:0/1039117415 >> 172.21.15.94:0/1039117415 conn(0x7f216c06a090 msgr2=0x7f216c13e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 1 -- 172.21.15.94:0/1039117415 shutdown_connections 2023-09-30T00:19:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:30.767+0000 7f217121b700 1 -- 172.21.15.94:0/1039117415 wait complete. 2023-09-30T00:19:33.781 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.905+0000 7fe2445dc700 1 Processor -- start 2023-09-30T00:19:33.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.905+0000 7fe2445dc700 1 -- start start 2023-09-30T00:19:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.906+0000 7fe2445dc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 0x7fe23c0a5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.906+0000 7fe2445dc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe23c0a6fc0 0x7fe23c148e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.906+0000 7fe2445dc700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23c0a6050 con 0x7fe23c0a4470 2023-09-30T00:19:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.906+0000 7fe2445dc700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23c0b4dd0 con 0x7fe23c0a6fc0 2023-09-30T00:19:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.906+0000 7fe2445dc700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c06ea50 con 0x7fe23c0a4840 2023-09-30T00:19:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.906+0000 7fe241b77700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 0x7fe23c0a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.906+0000 7fe241b77700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 0x7fe23c0a5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:35134/0 (socket says 172.21.15.94:35134) 2023-09-30T00:19:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.906+0000 7fe241b77700 1 -- 172.21.15.94:0/2055186843 learned_addr learned my addr 172.21.15.94:0/2055186843 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.906+0000 7fe241b77700 1 -- 172.21.15.94:0/2055186843 >> v2:172.21.15.121:3300/0 conn(0x7fe23c0a6fc0 msgr2=0x7fe23c148e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.906+0000 7fe241b77700 1 --2- 172.21.15.94:0/2055186843 >> v2:172.21.15.121:3300/0 conn(0x7fe23c0a6fc0 0x7fe23c148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.906+0000 7fe241b77700 1 -- 172.21.15.94:0/2055186843 >> v1:172.21.15.121:6789/0 conn(0x7fe23c0a4470 legacy=0x7fe23c0b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe241b77700 1 -- 172.21.15.94:0/2055186843 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23c0a5c60 con 0x7fe23c0a4840 2023-09-30T00:19:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe241b77700 1 --2- 172.21.15.94:0/2055186843 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 0x7fe23c0a5720 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe22c0099b0 tx=0x7fe22c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=271ea952ff83166b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe241376700 1 -- 172.21.15.94:0/2055186843 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c00b820 con 0x7fe23c0a4840 2023-09-30T00:19:33.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe241376700 1 -- 172.21.15.94:0/2055186843 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22c00b980 con 0x7fe23c0a4840 2023-09-30T00:19:33.910 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe241376700 1 -- 172.21.15.94:0/2055186843 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c00bd20 con 0x7fe23c0a4840 2023-09-30T00:19:33.912 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe2445dc700 1 -- 172.21.15.94:0/2055186843 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 msgr2=0x7fe23c0a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:33.912 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe2445dc700 1 --2- 172.21.15.94:0/2055186843 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 0x7fe23c0a5720 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe22c0099b0 tx=0x7fe22c009250 comp rx=0 tx=0).stop 2023-09-30T00:19:33.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe2445dc700 1 -- 172.21.15.94:0/2055186843 shutdown_connections 2023-09-30T00:19:33.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe2445dc700 1 --2- 172.21.15.94:0/2055186843 >> v2:172.21.15.121:3300/0 conn(0x7fe23c0a6fc0 0x7fe23c148e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:33.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe2445dc700 1 --2- 172.21.15.94:0/2055186843 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 0x7fe23c0a5720 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:33.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe2445dc700 1 -- 172.21.15.94:0/2055186843 >> 172.21.15.94:0/2055186843 conn(0x7fe23c06cbf0 msgr2=0x7fe23c06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:33.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe2445dc700 1 -- 172.21.15.94:0/2055186843 shutdown_connections 2023-09-30T00:19:33.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.907+0000 7fe2445dc700 1 -- 172.21.15.94:0/2055186843 wait complete. 2023-09-30T00:19:33.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.908+0000 7fe2445dc700 1 Processor -- start 2023-09-30T00:19:33.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.908+0000 7fe2445dc700 1 -- start start 2023-09-30T00:19:33.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe2445dc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23c0a4470 0x7fe23c0ad270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:33.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe2445dc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 0x7fe23c141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:33.913 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe2445dc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe23c0a6fc0 0x7fe23c14df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:33.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe2445dc700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c0a8da0 con 0x7fe23c0a6fc0 2023-09-30T00:19:33.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe2445dc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c14e960 con 0x7fe23c0a4470 2023-09-30T00:19:33.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe2445dc700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c14ec20 con 0x7fe23c0a4840 2023-09-30T00:19:33.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe241b77700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 0x7fe23c141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:33.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe241b77700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 0x7fe23c141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:35138/0 (socket says 172.21.15.94:35138) 2023-09-30T00:19:33.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe241b77700 1 -- 172.21.15.94:0/305750640 learned_addr learned my addr 172.21.15.94:0/305750640 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:33.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe242b79700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe23c0a6fc0 0x7fe23c14df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:33.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe242378700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23c0a4470 0x7fe23c0ad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:33.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe241b77700 1 -- 172.21.15.94:0/305750640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23c0a4470 msgr2=0x7fe23c0ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:33.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe241b77700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23c0a4470 0x7fe23c0ad270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:33.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe241b77700 1 -- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe23c0a6fc0 msgr2=0x7fe23c14df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:33.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe241b77700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe23c0a6fc0 0x7fe23c14df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:33.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe241b77700 1 -- 172.21.15.94:0/305750640 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c0094f0 con 0x7fe23c0a4840 2023-09-30T00:19:33.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe242378700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23c0a4470 0x7fe23c0ad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:19:33.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.909+0000 7fe241b77700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 0x7fe23c141ae0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe22c009290 tx=0x7fe22c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:33.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.910+0000 7fe2327fc700 1 -- 172.21.15.94:0/305750640 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c01a460 con 0x7fe23c0a4840 2023-09-30T00:19:33.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.910+0000 7fe2327fc700 1 -- 172.21.15.94:0/305750640 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22c01a5c0 con 0x7fe23c0a4840 2023-09-30T00:19:33.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.910+0000 7fe2445dc700 1 -- 172.21.15.94:0/305750640 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe23c142020 con 0x7fe23c0a4840 2023-09-30T00:19:33.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.910+0000 7fe2327fc700 1 -- 172.21.15.94:0/305750640 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c01a960 con 0x7fe23c0a4840 2023-09-30T00:19:33.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.910+0000 7fe2445dc700 1 -- 172.21.15.94:0/305750640 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe23c0b1c60 con 0x7fe23c0a4840 2023-09-30T00:19:33.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.911+0000 7fe2445dc700 1 -- 172.21.15.94:0/305750640 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe23c06fbd0 con 0x7fe23c0a4840 2023-09-30T00:19:33.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.911+0000 7fe2327fc700 1 -- 172.21.15.94:0/305750640 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe22c01ad10 con 0x7fe23c0a4840 2023-09-30T00:19:33.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.911+0000 7fe2327fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:33.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.912+0000 7fe2327fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:33.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.912+0000 7fe2327fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:33.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.912+0000 7fe2327fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:33.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.912+0000 7fe2327fc700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe220098780 0x7fe22009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:33.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.912+0000 7fe242378700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe220098780 0x7fe22009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:33.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.912+0000 7fe2327fc700 1 -- 172.21.15.94:0/305750640 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fe22c0c1e00 con 0x7fe23c0a4840 2023-09-30T00:19:33.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.912+0000 7fe242378700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe220098780 0x7fe22009ac30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe238007fd0 tx=0x7fe2380074f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:33.917 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:33.916+0000 7fe2327fc700 1 -- 172.21.15.94:0/305750640 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe22c08f4d0 con 0x7fe23c0a4840 2023-09-30T00:19:34.139 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.137+0000 7fe2445dc700 1 -- 172.21.15.94:0/305750640 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe23c06ecb0 con 0x7fe23c0a4840 2023-09-30T00:19:34.141 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.137+0000 7fe2327fc700 1 -- 172.21.15.94:0/305750640 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe22c02a100 con 0x7fe23c0a4840 2023-09-30T00:19:34.141 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:34.141 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:34.142 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 10 mgrc shutdown 2023-09-30T00:19:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 1 -- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe220098780 msgr2=0x7fe22009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe220098780 0x7fe22009ac30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe238007fd0 tx=0x7fe2380074f0 comp rx=0 tx=0).stop 2023-09-30T00:19:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 1 -- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 msgr2=0x7fe23c141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 0x7fe23c141ae0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe22c009290 tx=0x7fe22c016000 comp rx=0 tx=0).stop 2023-09-30T00:19:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 1 -- 172.21.15.94:0/305750640 shutdown_connections 2023-09-30T00:19:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe220098780 0x7fe22009ac30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe23c0a6fc0 0x7fe23c14df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe23c0a4470 0x7fe23c0ad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 1 --2- 172.21.15.94:0/305750640 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe23c0a4840 0x7fe23c141ae0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 1 -- 172.21.15.94:0/305750640 >> 172.21.15.94:0/305750640 conn(0x7fe23c06cbf0 msgr2=0x7fe23c06cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 1 -- 172.21.15.94:0/305750640 shutdown_connections 2023-09-30T00:19:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:34.139+0000 7fe2445dc700 1 -- 172.21.15.94:0/305750640 wait complete. 2023-09-30T00:19:37.153 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:37.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.273+0000 7f7bd6659700 1 Processor -- start 2023-09-30T00:19:37.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.273+0000 7f7bd6659700 1 -- start start 2023-09-30T00:19:37.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.274+0000 7f7bd6659700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7bd00b7160 0x7f7bd00a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:37.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.274+0000 7f7bd6659700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd013fd70 0x7f7bd01401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:37.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.274+0000 7f7bd6659700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd00a1310 con 0x7f7bd0070c50 2023-09-30T00:19:37.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.274+0000 7f7bd6659700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd0071020 con 0x7f7bd00b7160 2023-09-30T00:19:37.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.274+0000 7f7bd6659700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd00b76a0 con 0x7f7bd013fd70 2023-09-30T00:19:37.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.274+0000 7f7bd4bf6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd013fd70 0x7f7bd01401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:37.277 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.274+0000 7f7bd4bf6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd013fd70 0x7f7bd01401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:50342/0 (socket says 172.21.15.94:50342) 2023-09-30T00:19:37.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.274+0000 7f7bd4bf6700 1 -- 172.21.15.94:0/1968015752 learned_addr learned my addr 172.21.15.94:0/1968015752 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:37.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.274+0000 7f7bcf7fe700 1 --2- 172.21.15.94:0/1968015752 >> v2:172.21.15.121:3300/0 conn(0x7f7bd00b7160 0x7f7bd00a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:37.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.274+0000 7f7bd4bf6700 1 -- 172.21.15.94:0/1968015752 >> v2:172.21.15.121:3300/0 conn(0x7f7bd00b7160 msgr2=0x7f7bd00a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:37.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.274+0000 7f7bd4bf6700 1 --2- 172.21.15.94:0/1968015752 >> v2:172.21.15.121:3300/0 conn(0x7f7bd00b7160 0x7f7bd00a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:37.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.274+0000 7f7bd4bf6700 1 -- 172.21.15.94:0/1968015752 >> v1:172.21.15.121:6789/0 conn(0x7f7bd0070c50 legacy=0x7f7bd00a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:37.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bd4bf6700 1 -- 172.21.15.94:0/1968015752 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd00b7940 con 0x7f7bd013fd70 2023-09-30T00:19:37.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bd4bf6700 1 --2- 172.21.15.94:0/1968015752 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd013fd70 0x7f7bd01401d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009f20 tx=0x7f7bc8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4fcdee2f72a0194 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:37.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bceffd700 1 -- 172.21.15.94:0/1968015752 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc800e070 con 0x7f7bd013fd70 2023-09-30T00:19:37.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bceffd700 1 -- 172.21.15.94:0/1968015752 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc800bad0 con 0x7f7bd013fd70 2023-09-30T00:19:37.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bceffd700 1 -- 172.21.15.94:0/1968015752 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc80092e0 con 0x7f7bd013fd70 2023-09-30T00:19:37.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bd6659700 1 -- 172.21.15.94:0/1968015752 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd013fd70 msgr2=0x7f7bd01401d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:37.278 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bd6659700 1 --2- 172.21.15.94:0/1968015752 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd013fd70 0x7f7bd01401d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009f20 tx=0x7f7bc8009840 comp rx=0 tx=0).stop 2023-09-30T00:19:37.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bd6659700 1 -- 172.21.15.94:0/1968015752 shutdown_connections 2023-09-30T00:19:37.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bd6659700 1 --2- 172.21.15.94:0/1968015752 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd013fd70 0x7f7bd01401d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:37.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bd6659700 1 --2- 172.21.15.94:0/1968015752 >> v2:172.21.15.121:3300/0 conn(0x7f7bd00b7160 0x7f7bd00a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:37.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bd6659700 1 -- 172.21.15.94:0/1968015752 >> 172.21.15.94:0/1968015752 conn(0x7f7bd006a090 msgr2=0x7f7bd00702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:37.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bd6659700 1 -- 172.21.15.94:0/1968015752 shutdown_connections 2023-09-30T00:19:37.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.275+0000 7f7bd6659700 1 -- 172.21.15.94:0/1968015752 wait complete. 2023-09-30T00:19:37.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.276+0000 7f7bd6659700 1 Processor -- start 2023-09-30T00:19:37.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.277+0000 7f7bd6659700 1 -- start start 2023-09-30T00:19:37.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.277+0000 7f7bd6659700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd0070c50 0x7f7bd013c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:37.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.277+0000 7f7bd6659700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd00b7160 0x7f7bd0150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:37.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.277+0000 7f7bd6659700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7bd013fd70 0x7f7bd0150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:37.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.277+0000 7f7bd6659700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd0151630 con 0x7f7bd013fd70 2023-09-30T00:19:37.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.277+0000 7f7bd6659700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd013c870 con 0x7f7bd00b7160 2023-09-30T00:19:37.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.277+0000 7f7bd6659700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd0151180 con 0x7f7bd0070c50 2023-09-30T00:19:37.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.277+0000 7f7bcffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd0070c50 0x7f7bd013c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:37.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.277+0000 7f7bcffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd0070c50 0x7f7bd013c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:50346/0 (socket says 172.21.15.94:50346) 2023-09-30T00:19:37.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.277+0000 7f7bcffff700 1 -- 172.21.15.94:0/87576056 learned_addr learned my addr 172.21.15.94:0/87576056 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:37.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.277+0000 7f7bd4bf6700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7bd013fd70 0x7f7bd0150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:37.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.277+0000 7f7bcf7fe700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd00b7160 0x7f7bd0150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:37.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.278+0000 7f7bcffff700 1 -- 172.21.15.94:0/87576056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd00b7160 msgr2=0x7f7bd0150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:37.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.278+0000 7f7bcffff700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd00b7160 0x7f7bd0150700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:37.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.278+0000 7f7bcffff700 1 -- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7bd013fd70 msgr2=0x7f7bd0150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:37.283 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.278+0000 7f7bcffff700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7bd013fd70 0x7f7bd0150c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:37.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.278+0000 7f7bcffff700 1 -- 172.21.15.94:0/87576056 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc8009ae0 con 0x7f7bd0070c50 2023-09-30T00:19:37.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.278+0000 7f7bd4bf6700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7bd013fd70 0x7f7bd0150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:37.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.278+0000 7f7bcffff700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd0070c50 0x7f7bd013c010 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000c5d0 tx=0x7f7bc0007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:37.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.278+0000 7f7bb7fff700 1 -- 172.21.15.94:0/87576056 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0011590 con 0x7f7bd0070c50 2023-09-30T00:19:37.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.278+0000 7f7bb7fff700 1 -- 172.21.15.94:0/87576056 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc0011710 con 0x7f7bd0070c50 2023-09-30T00:19:37.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.278+0000 7f7bd6659700 1 -- 172.21.15.94:0/87576056 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd0151370 con 0x7f7bd0070c50 2023-09-30T00:19:37.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.278+0000 7f7bb7fff700 1 -- 172.21.15.94:0/87576056 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0011ad0 con 0x7f7bd0070c50 2023-09-30T00:19:37.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.278+0000 7f7bd6659700 1 -- 172.21.15.94:0/87576056 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd00a5b30 con 0x7f7bd0070c50 2023-09-30T00:19:37.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.279+0000 7f7bd6659700 1 -- 172.21.15.94:0/87576056 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd0071630 con 0x7f7bd0070c50 2023-09-30T00:19:37.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.280+0000 7f7bb7fff700 1 -- 172.21.15.94:0/87576056 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f7bc001c460 con 0x7f7bd0070c50 2023-09-30T00:19:37.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.280+0000 7f7bb7fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:37.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.280+0000 7f7bb7fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:37.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.280+0000 7f7bb7fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:37.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.280+0000 7f7bb7fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:37.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.280+0000 7f7bb7fff700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7bbc098780 0x7f7bbc09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:37.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.280+0000 7f7bcf7fe700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7bbc098780 0x7f7bbc09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:37.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.280+0000 7f7bb7fff700 1 -- 172.21.15.94:0/87576056 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f7bc00c00c0 con 0x7f7bd0070c50 2023-09-30T00:19:37.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.281+0000 7f7bcf7fe700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7bbc098780 0x7f7bbc09ac30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7bd014b090 tx=0x7f7bc4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:37.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.284+0000 7f7bb7fff700 1 -- 172.21.15.94:0/87576056 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7bc008d790 con 0x7f7bd0070c50 2023-09-30T00:19:37.507 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.506+0000 7f7bd6659700 1 -- 172.21.15.94:0/87576056 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7bd013f530 con 0x7f7bd0070c50 2023-09-30T00:19:37.507 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.506+0000 7f7bb7fff700 1 -- 172.21.15.94:0/87576056 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7bc0090db0 con 0x7f7bd0070c50 2023-09-30T00:19:37.508 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:37.508 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:37.510 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 10 mgrc shutdown 2023-09-30T00:19:37.510 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 1 -- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7bbc098780 msgr2=0x7f7bbc09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:37.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7bbc098780 0x7f7bbc09ac30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7bd014b090 tx=0x7f7bc4008040 comp rx=0 tx=0).stop 2023-09-30T00:19:37.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 1 -- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd0070c50 msgr2=0x7f7bd013c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:37.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd0070c50 0x7f7bd013c010 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000c5d0 tx=0x7f7bc0007f60 comp rx=0 tx=0).stop 2023-09-30T00:19:37.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 1 -- 172.21.15.94:0/87576056 shutdown_connections 2023-09-30T00:19:37.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7bbc098780 0x7f7bbc09ac30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:37.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7bd0070c50 0x7f7bd013c010 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:37.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7bd013fd70 0x7f7bd0150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:37.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 1 --2- 172.21.15.94:0/87576056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd00b7160 0x7f7bd0150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:37.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 1 -- 172.21.15.94:0/87576056 >> 172.21.15.94:0/87576056 conn(0x7f7bd006a090 msgr2=0x7f7bd01407b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:37.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 1 -- 172.21.15.94:0/87576056 shutdown_connections 2023-09-30T00:19:37.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:37.508+0000 7f7bd6659700 1 -- 172.21.15.94:0/87576056 wait complete. 2023-09-30T00:19:39.193 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:19:39.196 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:19:40.522 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:40.641 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.640+0000 7f5a8871d700 1 Processor -- start 2023-09-30T00:19:40.642 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.640+0000 7f5a8871d700 1 -- start start 2023-09-30T00:19:40.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a8871d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5a80070c50 0x7f5a800a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:40.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a8871d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5a8013fd70 0x7f5a80140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:40.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a8871d700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a800a1310 con 0x7f5a800b7160 2023-09-30T00:19:40.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a8871d700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a80071020 con 0x7f5a80070c50 2023-09-30T00:19:40.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a8871d700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a800b76a0 con 0x7f5a8013fd70 2023-09-30T00:19:40.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a864b9700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5a80070c50 0x7f5a800a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:40.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a864b9700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5a80070c50 0x7f5a800a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52326/0 (socket says 172.21.15.94:52326) 2023-09-30T00:19:40.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a864b9700 1 -- 172.21.15.94:0/2711965888 learned_addr learned my addr 172.21.15.94:0/2711965888 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:40.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a86cba700 1 --2- 172.21.15.94:0/2711965888 >> v2:172.21.15.121:3300/0 conn(0x7f5a8013fd70 0x7f5a80140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:40.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a864b9700 1 -- 172.21.15.94:0/2711965888 >> v2:172.21.15.121:3300/0 conn(0x7f5a8013fd70 msgr2=0x7f5a80140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:40.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a864b9700 1 --2- 172.21.15.94:0/2711965888 >> v2:172.21.15.121:3300/0 conn(0x7f5a8013fd70 0x7f5a80140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:40.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a864b9700 1 -- 172.21.15.94:0/2711965888 >> v1:172.21.15.121:6789/0 conn(0x7f5a800b7160 legacy=0x7f5a800a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:40.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.641+0000 7f5a864b9700 1 -- 172.21.15.94:0/2711965888 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a800b7940 con 0x7f5a80070c50 2023-09-30T00:19:40.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.642+0000 7f5a864b9700 1 --2- 172.21.15.94:0/2711965888 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a80070c50 0x7f5a800a0dd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5a800a31a0 tx=0x7f5a7c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b3cdcbaa4c6a69 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:40.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.642+0000 7f5a854b7700 1 -- 172.21.15.94:0/2711965888 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00e070 con 0x7f5a80070c50 2023-09-30T00:19:40.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.642+0000 7f5a854b7700 1 -- 172.21.15.94:0/2711965888 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c00bab0 con 0x7f5a80070c50 2023-09-30T00:19:40.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.642+0000 7f5a854b7700 1 -- 172.21.15.94:0/2711965888 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c0092e0 con 0x7f5a80070c50 2023-09-30T00:19:40.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.642+0000 7f5a8871d700 1 -- 172.21.15.94:0/2711965888 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a80070c50 msgr2=0x7f5a800a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:40.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.642+0000 7f5a8871d700 1 --2- 172.21.15.94:0/2711965888 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a80070c50 0x7f5a800a0dd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5a800a31a0 tx=0x7f5a7c009840 comp rx=0 tx=0).stop 2023-09-30T00:19:40.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.642+0000 7f5a8871d700 1 -- 172.21.15.94:0/2711965888 shutdown_connections 2023-09-30T00:19:40.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.642+0000 7f5a8871d700 1 --2- 172.21.15.94:0/2711965888 >> v2:172.21.15.121:3300/0 conn(0x7f5a8013fd70 0x7f5a80140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:40.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.642+0000 7f5a8871d700 1 --2- 172.21.15.94:0/2711965888 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a80070c50 0x7f5a800a0dd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:40.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.642+0000 7f5a8871d700 1 -- 172.21.15.94:0/2711965888 >> 172.21.15.94:0/2711965888 conn(0x7f5a8006a090 msgr2=0x7f5a800702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:40.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.642+0000 7f5a8871d700 1 -- 172.21.15.94:0/2711965888 shutdown_connections 2023-09-30T00:19:40.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.642+0000 7f5a8871d700 1 -- 172.21.15.94:0/2711965888 wait complete. 2023-09-30T00:19:40.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.643+0000 7f5a8871d700 1 Processor -- start 2023-09-30T00:19:40.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.643+0000 7f5a8871d700 1 -- start start 2023-09-30T00:19:40.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a8871d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a80070c50 0x7f5a800a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:40.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a8871d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5a800b7160 0x7f5a8009f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:40.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a8871d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a8013fd70 0x7f5a80142900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:40.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a8871d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a800a65e0 con 0x7f5a80070c50 2023-09-30T00:19:40.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a8871d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8013a6b0 con 0x7f5a8013fd70 2023-09-30T00:19:40.649 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a8871d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a800a58b0 con 0x7f5a800b7160 2023-09-30T00:19:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a864b9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a80070c50 0x7f5a800a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a864b9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a80070c50 0x7f5a800a4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52332/0 (socket says 172.21.15.94:52332) 2023-09-30T00:19:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a864b9700 1 -- 172.21.15.94:0/156699208 learned_addr learned my addr 172.21.15.94:0/156699208 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a85cb8700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5a800b7160 0x7f5a8009f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a86cba700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a8013fd70 0x7f5a80142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a864b9700 1 -- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5a800b7160 msgr2=0x7f5a8009f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a864b9700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5a800b7160 0x7f5a8009f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a864b9700 1 -- 172.21.15.94:0/156699208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a8013fd70 msgr2=0x7f5a80142900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a864b9700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a8013fd70 0x7f5a80142900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a864b9700 1 -- 172.21.15.94:0/156699208 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7c009ae0 con 0x7f5a80070c50 2023-09-30T00:19:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a86cba700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a8013fd70 0x7f5a80142900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:19:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a864b9700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a80070c50 0x7f5a800a4e50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5a800a97e0 tx=0x7f5a7c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a767fc700 1 -- 172.21.15.94:0/156699208 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c008a80 con 0x7f5a80070c50 2023-09-30T00:19:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a767fc700 1 -- 172.21.15.94:0/156699208 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c008c00 con 0x7f5a80070c50 2023-09-30T00:19:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a8871d700 1 -- 172.21.15.94:0/156699208 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a800a5bd0 con 0x7f5a80070c50 2023-09-30T00:19:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.644+0000 7f5a767fc700 1 -- 172.21.15.94:0/156699208 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c026460 con 0x7f5a80070c50 2023-09-30T00:19:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.645+0000 7f5a8871d700 1 -- 172.21.15.94:0/156699208 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8009bc00 con 0x7f5a80070c50 2023-09-30T00:19:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.645+0000 7f5a8871d700 1 -- 172.21.15.94:0/156699208 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8009bdf0 con 0x7f5a80070c50 2023-09-30T00:19:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.646+0000 7f5a767fc700 1 -- 172.21.15.94:0/156699208 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5a7c003040 con 0x7f5a80070c50 2023-09-30T00:19:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.646+0000 7f5a767fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.647+0000 7f5a767fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.647+0000 7f5a767fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.647+0000 7f5a767fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.647+0000 7f5a767fc700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5a64098780 0x7f5a6409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.647+0000 7f5a767fc700 1 -- 172.21.15.94:0/156699208 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f5a7c0c25d0 con 0x7f5a80070c50 2023-09-30T00:19:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.647+0000 7f5a85cb8700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5a64098780 0x7f5a6409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.647+0000 7f5a85cb8700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5a64098780 0x7f5a6409ac30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5a80144c20 tx=0x7f5a70009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.650+0000 7f5a767fc700 1 -- 172.21.15.94:0/156699208 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a7c08fca0 con 0x7f5a80070c50 2023-09-30T00:19:40.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.871+0000 7f5a8871d700 1 -- 172.21.15.94:0/156699208 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5a8013a880 con 0x7f5a80070c50 2023-09-30T00:19:40.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.872+0000 7f5a767fc700 1 -- 172.21.15.94:0/156699208 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5a7c0932c0 con 0x7f5a80070c50 2023-09-30T00:19:40.873 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:40.874 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:40.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 10 mgrc shutdown 2023-09-30T00:19:40.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 1 -- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5a64098780 msgr2=0x7f5a6409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:40.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5a64098780 0x7f5a6409ac30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5a80144c20 tx=0x7f5a70009040 comp rx=0 tx=0).stop 2023-09-30T00:19:40.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 1 -- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a80070c50 msgr2=0x7f5a800a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:40.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a80070c50 0x7f5a800a4e50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5a800a97e0 tx=0x7f5a7c018000 comp rx=0 tx=0).stop 2023-09-30T00:19:40.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 1 -- 172.21.15.94:0/156699208 shutdown_connections 2023-09-30T00:19:40.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5a64098780 0x7f5a6409ac30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:40.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a8013fd70 0x7f5a80142900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:40.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a80070c50 0x7f5a800a4e50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:40.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 1 --2- 172.21.15.94:0/156699208 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5a800b7160 0x7f5a8009f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:40.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 1 -- 172.21.15.94:0/156699208 >> 172.21.15.94:0/156699208 conn(0x7f5a8006a090 msgr2=0x7f5a800702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:40.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 1 -- 172.21.15.94:0/156699208 shutdown_connections 2023-09-30T00:19:40.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:40.874+0000 7f5a8871d700 1 -- 172.21.15.94:0/156699208 wait complete. 2023-09-30T00:19:43.888 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:44.014 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.012+0000 7f93a33f5700 1 Processor -- start 2023-09-30T00:19:44.014 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.012+0000 7f93a33f5700 1 -- start start 2023-09-30T00:19:44.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.013+0000 7f93a33f5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 0x7f939c0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:44.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.013+0000 7f93a33f5700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f939c0a1310 con 0x7f939c070c50 2023-09-30T00:19:44.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.013+0000 7f93a33f5700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f939c071020 con 0x7f939c13fd70 2023-09-30T00:19:44.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.013+0000 7f93a33f5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939c0b76a0 con 0x7f939c0b7160 2023-09-30T00:19:44.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.013+0000 7f93a1191700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f939c070c50 0x7f939c0a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58500/0 (socket says 172.21.15.94:58500) 2023-09-30T00:19:44.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.013+0000 7f93a1191700 1 -- 172.21.15.94:0/3377596352 learned_addr learned my addr 172.21.15.94:0/3377596352 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:19:44.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.013+0000 7f93a0990700 1 --2- 172.21.15.94:0/3377596352 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 0x7f939c0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.013+0000 7f9393fff700 1 -- 172.21.15.94:0/3377596352 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748591216 0 0) 0x7f9398001110 con 0x7f939c070c50 2023-09-30T00:19:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.013+0000 7f9393fff700 1 -- 172.21.15.94:0/3377596352 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9388003570 con 0x7f939c070c50 2023-09-30T00:19:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.013+0000 7f93a0990700 1 -- 172.21.15.94:0/3377596352 >> v1:172.21.15.121:6789/0 conn(0x7f939c13fd70 legacy=0x7f939c14b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.013+0000 7f93a0990700 1 -- 172.21.15.94:0/3377596352 >> v1:172.21.15.94:6789/0 conn(0x7f939c070c50 legacy=0x7f939c0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.013+0000 7f93a0990700 1 -- 172.21.15.94:0/3377596352 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939c0b7940 con 0x7f939c0b7160 2023-09-30T00:19:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.014+0000 7f93a0990700 1 --2- 172.21.15.94:0/3377596352 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 0x7f939c0a0dd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f938c009910 tx=0x7f938c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69b784fe781c61c0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.014+0000 7f9393fff700 1 -- 172.21.15.94:0/3377596352 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c00b6d0 con 0x7f939c0b7160 2023-09-30T00:19:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.014+0000 7f9393fff700 1 -- 172.21.15.94:0/3377596352 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c00b830 con 0x7f939c0b7160 2023-09-30T00:19:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.014+0000 7f9393fff700 1 -- 172.21.15.94:0/3377596352 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c00bbf0 con 0x7f939c0b7160 2023-09-30T00:19:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.014+0000 7f93a33f5700 1 -- 172.21.15.94:0/3377596352 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 msgr2=0x7f939c0a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.014+0000 7f93a33f5700 1 --2- 172.21.15.94:0/3377596352 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 0x7f939c0a0dd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f938c009910 tx=0x7f938c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:19:44.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.014+0000 7f93a33f5700 1 -- 172.21.15.94:0/3377596352 shutdown_connections 2023-09-30T00:19:44.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.014+0000 7f93a33f5700 1 --2- 172.21.15.94:0/3377596352 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 0x7f939c0a0dd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:44.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.014+0000 7f93a33f5700 1 -- 172.21.15.94:0/3377596352 >> 172.21.15.94:0/3377596352 conn(0x7f939c06a090 msgr2=0x7f939c0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:44.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.014+0000 7f93a33f5700 1 -- 172.21.15.94:0/3377596352 shutdown_connections 2023-09-30T00:19:44.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.014+0000 7f93a33f5700 1 -- 172.21.15.94:0/3377596352 wait complete. 2023-09-30T00:19:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.015+0000 7f93a33f5700 1 Processor -- start 2023-09-30T00:19:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.015+0000 7f93a33f5700 1 -- start start 2023-09-30T00:19:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.016+0000 7f93a33f5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c070c50 0x7f939c13c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.016+0000 7f93a33f5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 0x7f939c0ad680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.016+0000 7f93a33f5700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939c13fd70 0x7f939c143e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.016+0000 7f93a33f5700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c1516f0 con 0x7f939c13fd70 2023-09-30T00:19:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.016+0000 7f93a33f5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c0adbc0 con 0x7f939c070c50 2023-09-30T00:19:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.016+0000 7f93a33f5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939c0aad20 con 0x7f939c0b7160 2023-09-30T00:19:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.016+0000 7f93a0990700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 0x7f939c0ad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.016+0000 7f93a0990700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 0x7f939c0ad680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:50362/0 (socket says 172.21.15.94:50362) 2023-09-30T00:19:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.016+0000 7f93a1992700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939c13fd70 0x7f939c143e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.016+0000 7f93a0990700 1 -- 172.21.15.94:0/192670856 learned_addr learned my addr 172.21.15.94:0/192670856 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.016+0000 7f93a1191700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c070c50 0x7f939c13c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.017+0000 7f93a0990700 1 -- 172.21.15.94:0/192670856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c070c50 msgr2=0x7f939c13c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.017+0000 7f93a0990700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c070c50 0x7f939c13c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.017+0000 7f93a0990700 1 -- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939c13fd70 msgr2=0x7f939c143e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.017+0000 7f93a0990700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939c13fd70 0x7f939c143e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.017+0000 7f93a0990700 1 -- 172.21.15.94:0/192670856 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c009420 con 0x7f939c0b7160 2023-09-30T00:19:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.017+0000 7f93a1992700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939c13fd70 0x7f939c143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.017+0000 7f93a0990700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 0x7f939c0ad680 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f938c0091e0 tx=0x7f938c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.017+0000 7f93917fa700 1 -- 172.21.15.94:0/192670856 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c00b9a0 con 0x7f939c0b7160 2023-09-30T00:19:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.017+0000 7f93917fa700 1 -- 172.21.15.94:0/192670856 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c020070 con 0x7f939c0b7160 2023-09-30T00:19:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.017+0000 7f93a33f5700 1 -- 172.21.15.94:0/192670856 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939c0aa8f0 con 0x7f939c0b7160 2023-09-30T00:19:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.017+0000 7f93917fa700 1 -- 172.21.15.94:0/192670856 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c01b450 con 0x7f939c0b7160 2023-09-30T00:19:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.017+0000 7f93a33f5700 1 -- 172.21.15.94:0/192670856 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939c1505f0 con 0x7f939c0b7160 2023-09-30T00:19:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.018+0000 7f93a33f5700 1 -- 172.21.15.94:0/192670856 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939c071630 con 0x7f939c0b7160 2023-09-30T00:19:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.019+0000 7f93917fa700 1 -- 172.21.15.94:0/192670856 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f938c022030 con 0x7f939c0b7160 2023-09-30T00:19:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.019+0000 7f93917fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.019+0000 7f93917fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.019+0000 7f93917fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.019+0000 7f93917fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.019+0000 7f93917fa700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9380098780 0x7f938009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.019+0000 7f93917fa700 1 -- 172.21.15.94:0/192670856 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f938c0c21b0 con 0x7f939c0b7160 2023-09-30T00:19:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.019+0000 7f93a1191700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9380098780 0x7f938009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.020+0000 7f93a1191700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9380098780 0x7f938009ac30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9398001270 tx=0x7f9398009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:44.025 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.024+0000 7f93917fa700 1 -- 172.21.15.94:0/192670856 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f938c08f880 con 0x7f939c0b7160 2023-09-30T00:19:44.247 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.246+0000 7f93a33f5700 1 -- 172.21.15.94:0/192670856 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f939c071630 con 0x7f939c0b7160 2023-09-30T00:19:44.247 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.246+0000 7f93917fa700 1 -- 172.21.15.94:0/192670856 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f938c092ea0 con 0x7f939c0b7160 2023-09-30T00:19:44.248 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:44.248 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:44.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 10 mgrc shutdown 2023-09-30T00:19:44.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 1 -- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9380098780 msgr2=0x7f938009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:44.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9380098780 0x7f938009ac30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9398001270 tx=0x7f9398009040 comp rx=0 tx=0).stop 2023-09-30T00:19:44.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 1 -- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 msgr2=0x7f939c0ad680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:44.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 0x7f939c0ad680 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f938c0091e0 tx=0x7f938c016000 comp rx=0 tx=0).stop 2023-09-30T00:19:44.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 1 -- 172.21.15.94:0/192670856 shutdown_connections 2023-09-30T00:19:44.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9380098780 0x7f938009ac30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:44.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939c13fd70 0x7f939c143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:44.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c070c50 0x7f939c13c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:44.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 1 --2- 172.21.15.94:0/192670856 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f939c0b7160 0x7f939c0ad680 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:44.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 1 -- 172.21.15.94:0/192670856 >> 172.21.15.94:0/192670856 conn(0x7f939c06a090 msgr2=0x7f939c140fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 1 -- 172.21.15.94:0/192670856 shutdown_connections 2023-09-30T00:19:44.251 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:44.248+0000 7f93a33f5700 1 -- 172.21.15.94:0/192670856 wait complete. 2023-09-30T00:19:47.262 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:47.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.389+0000 7f5c76978700 1 Processor -- start 2023-09-30T00:19:47.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.389+0000 7f5c76978700 1 -- start start 2023-09-30T00:19:47.392 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.390+0000 7f5c76978700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5c700a4470 0x7f5c700b5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:47.392 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.390+0000 7f5c76978700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 0x7f5c70148e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.390+0000 7f5c76978700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c700a6050 con 0x7f5c700a4840 2023-09-30T00:19:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.390+0000 7f5c76978700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c700a5e30 con 0x7f5c700a4470 2023-09-30T00:19:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.390+0000 7f5c76978700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700b4dd0 con 0x7f5c700a6fc0 2023-09-30T00:19:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.390+0000 7f5c6ffff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5c700a4470 0x7f5c700b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.390+0000 7f5c74f15700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 0x7f5c70148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.390+0000 7f5c74f15700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 0x7f5c70148e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:45246/0 (socket says 172.21.15.94:45246) 2023-09-30T00:19:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.390+0000 7f5c6ffff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5c700a4470 0x7f5c700b5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56758/0 (socket says 172.21.15.94:56758) 2023-09-30T00:19:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.390+0000 7f5c74f15700 1 -- 172.21.15.94:0/2390912893 learned_addr learned my addr 172.21.15.94:0/2390912893 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c74f15700 1 -- 172.21.15.94:0/2390912893 >> v2:172.21.15.94:3300/0 conn(0x7f5c700a4470 msgr2=0x7f5c700b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c74f15700 1 --2- 172.21.15.94:0/2390912893 >> v2:172.21.15.94:3300/0 conn(0x7f5c700a4470 0x7f5c700b5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c74f15700 1 -- 172.21.15.94:0/2390912893 >> v1:172.21.15.94:6789/0 conn(0x7f5c700a4840 legacy=0x7f5c700a5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c74f15700 1 -- 172.21.15.94:0/2390912893 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7006ea50 con 0x7f5c700a6fc0 2023-09-30T00:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c6ffff700 1 --2- 172.21.15.94:0/2390912893 >> v2:172.21.15.94:3300/0 conn(0x7f5c700a4470 0x7f5c700b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c74f15700 1 --2- 172.21.15.94:0/2390912893 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 0x7f5c70148e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5c68009f20 tx=0x7f5c68009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd6899b146e51149 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c6effd700 1 -- 172.21.15.94:0/2390912893 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6800e070 con 0x7f5c700a6fc0 2023-09-30T00:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c6effd700 1 -- 172.21.15.94:0/2390912893 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6800b9d0 con 0x7f5c700a6fc0 2023-09-30T00:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c6effd700 1 -- 172.21.15.94:0/2390912893 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c680092e0 con 0x7f5c700a6fc0 2023-09-30T00:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c76978700 1 -- 172.21.15.94:0/2390912893 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 msgr2=0x7f5c70148e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c76978700 1 --2- 172.21.15.94:0/2390912893 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 0x7f5c70148e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5c68009f20 tx=0x7f5c68009840 comp rx=0 tx=0).stop 2023-09-30T00:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c76978700 1 -- 172.21.15.94:0/2390912893 shutdown_connections 2023-09-30T00:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c76978700 1 --2- 172.21.15.94:0/2390912893 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 0x7f5c70148e30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c76978700 1 --2- 172.21.15.94:0/2390912893 >> v2:172.21.15.94:3300/0 conn(0x7f5c700a4470 0x7f5c700b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c76978700 1 -- 172.21.15.94:0/2390912893 >> 172.21.15.94:0/2390912893 conn(0x7f5c7006cbf0 msgr2=0x7f5c7006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c76978700 1 -- 172.21.15.94:0/2390912893 shutdown_connections 2023-09-30T00:19:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.391+0000 7f5c76978700 1 -- 172.21.15.94:0/2390912893 wait complete. 2023-09-30T00:19:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.392+0000 7f5c76978700 1 Processor -- start 2023-09-30T00:19:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c76978700 1 -- start start 2023-09-30T00:19:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c76978700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c700a4470 0x7f5c70140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c76978700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c700a4840 0x7f5c700a8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c76978700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 0x7f5c700a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c76978700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700ab840 con 0x7f5c700a4470 2023-09-30T00:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c76978700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70141ab0 con 0x7f5c700a4840 2023-09-30T00:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c76978700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700a99d0 con 0x7f5c700a6fc0 2023-09-30T00:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c74f15700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 0x7f5c700a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c700a4470 0x7f5c70140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c74f15700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 0x7f5c700a92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:45252/0 (socket says 172.21.15.94:45252) 2023-09-30T00:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c74f15700 1 -- 172.21.15.94:0/1264240896 learned_addr learned my addr 172.21.15.94:0/1264240896 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c74f15700 1 -- 172.21.15.94:0/1264240896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c700a4840 msgr2=0x7f5c700a8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c74f15700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c700a4840 0x7f5c700a8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c74f15700 1 -- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c700a4470 msgr2=0x7f5c70140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c74f15700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c700a4470 0x7f5c70140bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c74f15700 1 -- 172.21.15.94:0/1264240896 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c60009420 con 0x7f5c700a6fc0 2023-09-30T00:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.393+0000 7f5c6ffff700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c700a4470 0x7f5c70140bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.394+0000 7f5c74f15700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 0x7f5c700a92e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800b610 tx=0x7f5c68018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.394+0000 7f5c57fff700 1 -- 172.21.15.94:0/1264240896 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c680094e0 con 0x7f5c700a6fc0 2023-09-30T00:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.394+0000 7f5c57fff700 1 -- 172.21.15.94:0/1264240896 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c68007b70 con 0x7f5c700a6fc0 2023-09-30T00:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.394+0000 7f5c76978700 1 -- 172.21.15.94:0/1264240896 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c68009ae0 con 0x7f5c700a6fc0 2023-09-30T00:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.394+0000 7f5c57fff700 1 -- 172.21.15.94:0/1264240896 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6801e430 con 0x7f5c700a6fc0 2023-09-30T00:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.394+0000 7f5c76978700 1 -- 172.21.15.94:0/1264240896 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7013f240 con 0x7f5c700a6fc0 2023-09-30T00:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.395+0000 7f5c76978700 1 -- 172.21.15.94:0/1264240896 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7006fbd0 con 0x7f5c700a6fc0 2023-09-30T00:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.396+0000 7f5c57fff700 1 -- 172.21.15.94:0/1264240896 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5c68003040 con 0x7f5c700a6fc0 2023-09-30T00:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.396+0000 7f5c57fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.396+0000 7f5c57fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.396+0000 7f5c57fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.396+0000 7f5c57fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.396+0000 7f5c57fff700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5c5c098780 0x7f5c5c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.396+0000 7f5c6ffff700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5c5c098780 0x7f5c5c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.396+0000 7f5c57fff700 1 -- 172.21.15.94:0/1264240896 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f5c680c26c0 con 0x7f5c700a6fc0 2023-09-30T00:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.397+0000 7f5c6ffff700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5c5c098780 0x7f5c5c09ac30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5c60011560 tx=0x7f5c600091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.400+0000 7f5c57fff700 1 -- 172.21.15.94:0/1264240896 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5c6805e5a0 con 0x7f5c700a6fc0 2023-09-30T00:19:47.623 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.623+0000 7f5c76978700 1 -- 172.21.15.94:0/1264240896 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5c7006e570 con 0x7f5c700a6fc0 2023-09-30T00:19:47.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.623+0000 7f5c57fff700 1 -- 172.21.15.94:0/1264240896 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5c6808fd90 con 0x7f5c700a6fc0 2023-09-30T00:19:47.624 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:47.624 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:47.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.625+0000 7f5c76978700 10 mgrc shutdown 2023-09-30T00:19:47.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.626+0000 7f5c76978700 1 -- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5c5c098780 msgr2=0x7f5c5c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:47.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.626+0000 7f5c76978700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5c5c098780 0x7f5c5c09ac30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5c60011560 tx=0x7f5c600091a0 comp rx=0 tx=0).stop 2023-09-30T00:19:47.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.626+0000 7f5c76978700 1 -- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 msgr2=0x7f5c700a92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:47.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.626+0000 7f5c76978700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 0x7f5c700a92e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800b610 tx=0x7f5c68018000 comp rx=0 tx=0).stop 2023-09-30T00:19:47.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.626+0000 7f5c76978700 1 -- 172.21.15.94:0/1264240896 shutdown_connections 2023-09-30T00:19:47.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.626+0000 7f5c76978700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5c5c098780 0x7f5c5c09ac30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:47.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.626+0000 7f5c76978700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5c700a6fc0 0x7f5c700a92e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:47.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.626+0000 7f5c76978700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c700a4470 0x7f5c70140bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:47.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.626+0000 7f5c76978700 1 --2- 172.21.15.94:0/1264240896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c700a4840 0x7f5c700a8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:47.628 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.626+0000 7f5c76978700 1 -- 172.21.15.94:0/1264240896 >> 172.21.15.94:0/1264240896 conn(0x7f5c7006cbf0 msgr2=0x7f5c7013c090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:47.628 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.626+0000 7f5c76978700 1 -- 172.21.15.94:0/1264240896 shutdown_connections 2023-09-30T00:19:47.628 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:47.626+0000 7f5c76978700 1 -- 172.21.15.94:0/1264240896 wait complete. 2023-09-30T00:19:50.642 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:50.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.763+0000 7f3b3fe7e700 1 Processor -- start 2023-09-30T00:19:50.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.763+0000 7f3b3fe7e700 1 -- start start 2023-09-30T00:19:50.766 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3fe7e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3b380b7160 0x7f3b380a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:50.766 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3fe7e700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3b3813fd70 0x7f3b38140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:50.766 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3fe7e700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b380a1310 con 0x7f3b38070c50 2023-09-30T00:19:50.766 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3fe7e700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b38071020 con 0x7f3b3813fd70 2023-09-30T00:19:50.766 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3fe7e700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b380b76a0 con 0x7f3b380b7160 2023-09-30T00:19:50.766 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3e41b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3b3813fd70 0x7f3b38140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:50.766 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3e41b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3b3813fd70 0x7f3b38140180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56774/0 (socket says 172.21.15.94:56774) 2023-09-30T00:19:50.766 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3e41b700 1 -- 172.21.15.94:0/1125869046 learned_addr learned my addr 172.21.15.94:0/1125869046 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:50.767 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3e41b700 1 -- 172.21.15.94:0/1125869046 >> v2:172.21.15.121:3300/0 conn(0x7f3b380b7160 msgr2=0x7f3b380a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:50.767 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3d419700 1 --2- 172.21.15.94:0/1125869046 >> v2:172.21.15.121:3300/0 conn(0x7f3b380b7160 0x7f3b380a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:50.767 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3e41b700 1 --2- 172.21.15.94:0/1125869046 >> v2:172.21.15.121:3300/0 conn(0x7f3b380b7160 0x7f3b380a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:50.767 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3e41b700 1 -- 172.21.15.94:0/1125869046 >> v1:172.21.15.121:6789/0 conn(0x7f3b38070c50 legacy=0x7f3b380a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:50.767 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3e41b700 1 -- 172.21.15.94:0/1125869046 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b380b7940 con 0x7f3b3813fd70 2023-09-30T00:19:50.767 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.764+0000 7f3b3e41b700 1 --2- 172.21.15.94:0/1125869046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b3813fd70 0x7f3b38140180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3b3806a880 tx=0x7f3b30009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d87827e00f3eb04 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:50.767 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.765+0000 7f3b3cc18700 1 -- 172.21.15.94:0/1125869046 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000e070 con 0x7f3b3813fd70 2023-09-30T00:19:50.767 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.765+0000 7f3b3cc18700 1 -- 172.21.15.94:0/1125869046 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3000baa0 con 0x7f3b3813fd70 2023-09-30T00:19:50.767 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.765+0000 7f3b3cc18700 1 -- 172.21.15.94:0/1125869046 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b300092e0 con 0x7f3b3813fd70 2023-09-30T00:19:50.767 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.765+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1125869046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b3813fd70 msgr2=0x7f3b38140180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:50.767 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.765+0000 7f3b3fe7e700 1 --2- 172.21.15.94:0/1125869046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b3813fd70 0x7f3b38140180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3b3806a880 tx=0x7f3b30009840 comp rx=0 tx=0).stop 2023-09-30T00:19:50.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.765+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1125869046 shutdown_connections 2023-09-30T00:19:50.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.765+0000 7f3b3fe7e700 1 --2- 172.21.15.94:0/1125869046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b3813fd70 0x7f3b38140180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:50.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.765+0000 7f3b3fe7e700 1 --2- 172.21.15.94:0/1125869046 >> v2:172.21.15.121:3300/0 conn(0x7f3b380b7160 0x7f3b380a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:50.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.765+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1125869046 >> 172.21.15.94:0/1125869046 conn(0x7f3b3806a090 msgr2=0x7f3b380702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:50.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.765+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1125869046 shutdown_connections 2023-09-30T00:19:50.768 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.765+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1125869046 wait complete. 2023-09-30T00:19:50.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.766+0000 7f3b3fe7e700 1 Processor -- start 2023-09-30T00:19:50.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.766+0000 7f3b3fe7e700 1 -- start start 2023-09-30T00:19:50.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3fe7e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b38070c50 0x7f3b38142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:50.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3fe7e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b380b7160 0x7f3b38139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:50.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3fe7e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3b3813fd70 0x7f3b380a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:50.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3fe7e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b381432b0 con 0x7f3b380b7160 2023-09-30T00:19:50.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3fe7e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b38143550 con 0x7f3b38070c50 2023-09-30T00:19:50.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3fe7e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b38144a00 con 0x7f3b3813fd70 2023-09-30T00:19:50.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3d419700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b380b7160 0x7f3b38139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:50.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3d419700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b380b7160 0x7f3b38139aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56782/0 (socket says 172.21.15.94:56782) 2023-09-30T00:19:50.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3d419700 1 -- 172.21.15.94:0/1495059797 learned_addr learned my addr 172.21.15.94:0/1495059797 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:50.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3e41b700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3b3813fd70 0x7f3b380a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:50.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3dc1a700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b38070c50 0x7f3b38142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:50.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3d419700 1 -- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3b3813fd70 msgr2=0x7f3b380a44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:50.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3d419700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3b3813fd70 0x7f3b380a44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:50.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3d419700 1 -- 172.21.15.94:0/1495059797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b38070c50 msgr2=0x7f3b38142bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:50.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3d419700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b38070c50 0x7f3b38142bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:50.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3d419700 1 -- 172.21.15.94:0/1495059797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b30009ae0 con 0x7f3b380b7160 2023-09-30T00:19:50.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3e41b700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3b3813fd70 0x7f3b380a44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:50.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3d419700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b380b7160 0x7f3b38139aa0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800c7d0 tx=0x7f3b28008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:50.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b2dffb700 1 -- 172.21.15.94:0/1495059797 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28011600 con 0x7f3b380b7160 2023-09-30T00:19:50.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b2dffb700 1 -- 172.21.15.94:0/1495059797 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b28011760 con 0x7f3b380b7160 2023-09-30T00:19:50.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1495059797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b380a4070 con 0x7f3b380b7160 2023-09-30T00:19:50.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.767+0000 7f3b2dffb700 1 -- 172.21.15.94:0/1495059797 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28011b20 con 0x7f3b380b7160 2023-09-30T00:19:50.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.768+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1495059797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b380a5ac0 con 0x7f3b380b7160 2023-09-30T00:19:50.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.768+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1495059797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b380a42b0 con 0x7f3b380b7160 2023-09-30T00:19:50.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.770+0000 7f3b2dffb700 1 -- 172.21.15.94:0/1495059797 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f3b280193f0 con 0x7f3b380b7160 2023-09-30T00:19:50.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.770+0000 7f3b2dffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:50.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.770+0000 7f3b2dffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:50.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.770+0000 7f3b2dffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:50.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.770+0000 7f3b2dffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:50.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.770+0000 7f3b2dffb700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3b1c098800 0x7f3b1c09acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:50.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.770+0000 7f3b3dc1a700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3b1c098800 0x7f3b1c09acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:50.774 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.770+0000 7f3b2dffb700 1 -- 172.21.15.94:0/1495059797 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f3b280c0e10 con 0x7f3b380b7160 2023-09-30T00:19:50.774 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.770+0000 7f3b3dc1a700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3b1c098800 0x7f3b1c09acb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3b34003a30 tx=0x7f3b34007ea0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:50.774 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.773+0000 7f3b2dffb700 1 -- 172.21.15.94:0/1495059797 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3b280515a0 con 0x7f3b380b7160 2023-09-30T00:19:50.998 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.998+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1495059797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3b380a0800 con 0x7f3b380b7160 2023-09-30T00:19:50.998 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:50.998+0000 7f3b2dffb700 1 -- 172.21.15.94:0/1495059797 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3b28091bb0 con 0x7f3b380b7160 2023-09-30T00:19:50.999 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:50.999 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:51.001 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 10 mgrc shutdown 2023-09-30T00:19:51.001 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3b1c098800 msgr2=0x7f3b1c09acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:51.001 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3b1c098800 0x7f3b1c09acb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3b34003a30 tx=0x7f3b34007ea0 comp rx=0 tx=0).stop 2023-09-30T00:19:51.001 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b380b7160 msgr2=0x7f3b38139aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:51.001 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b380b7160 0x7f3b38139aa0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800c7d0 tx=0x7f3b28008040 comp rx=0 tx=0).stop 2023-09-30T00:19:51.001 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1495059797 shutdown_connections 2023-09-30T00:19:51.001 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3b1c098800 0x7f3b1c09acb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:51.001 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3b3813fd70 0x7f3b380a44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:51.001 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b38070c50 0x7f3b38142bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:51.001 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 1 --2- 172.21.15.94:0/1495059797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b380b7160 0x7f3b38139aa0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:51.001 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1495059797 >> 172.21.15.94:0/1495059797 conn(0x7f3b3806a090 msgr2=0x7f3b38067cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:51.002 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1495059797 shutdown_connections 2023-09-30T00:19:51.002 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:51.000+0000 7f3b3fe7e700 1 -- 172.21.15.94:0/1495059797 wait complete. 2023-09-30T00:19:54.013 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:54.151 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.149+0000 7f2f0e407700 1 Processor -- start 2023-09-30T00:19:54.151 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.149+0000 7f2f0e407700 1 -- start start 2023-09-30T00:19:54.151 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.149+0000 7f2f0e407700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 0x7f2f080a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:54.151 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.149+0000 7f2f0e407700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f080a1310 con 0x7f2f0813fd70 2023-09-30T00:19:54.151 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.149+0000 7f2f0e407700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f08071020 con 0x7f2f08070c50 2023-09-30T00:19:54.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.149+0000 7f2f0e407700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f080b76a0 con 0x7f2f080b7160 2023-09-30T00:19:54.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.149+0000 7f2f077fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 0x7f2f080a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:54.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.149+0000 7f2f0c9a4700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2f0813fd70 0x7f2f0814b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38798/0 (socket says 172.21.15.94:38798) 2023-09-30T00:19:54.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.149+0000 7f2f0c9a4700 1 -- 172.21.15.94:0/2194757847 learned_addr learned my addr 172.21.15.94:0/2194757847 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:19:54.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.150+0000 7f2f06ffd700 1 -- 172.21.15.94:0/2194757847 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289603452 0 0) 0x7f2f00001490 con 0x7f2f0813fd70 2023-09-30T00:19:54.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.150+0000 7f2f077fe700 1 -- 172.21.15.94:0/2194757847 >> v1:172.21.15.121:6789/0 conn(0x7f2f08070c50 legacy=0x7f2f080a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:19:54.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.150+0000 7f2f077fe700 1 -- 172.21.15.94:0/2194757847 >> v1:172.21.15.94:6789/0 conn(0x7f2f0813fd70 legacy=0x7f2f0814b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:54.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.150+0000 7f2f077fe700 1 -- 172.21.15.94:0/2194757847 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f080b7940 con 0x7f2f080b7160 2023-09-30T00:19:54.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.150+0000 7f2f077fe700 1 --2- 172.21.15.94:0/2194757847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 0x7f2f080a0dd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009910 tx=0x7f2ef80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a7f2a81db837e78 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:54.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.150+0000 7f2f06ffd700 1 -- 172.21.15.94:0/2194757847 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef800b8b0 con 0x7f2f080b7160 2023-09-30T00:19:54.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.150+0000 7f2f06ffd700 1 -- 172.21.15.94:0/2194757847 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef800ba10 con 0x7f2f080b7160 2023-09-30T00:19:54.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.150+0000 7f2f06ffd700 1 -- 172.21.15.94:0/2194757847 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef800bdb0 con 0x7f2f080b7160 2023-09-30T00:19:54.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.150+0000 7f2f0e407700 1 -- 172.21.15.94:0/2194757847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 msgr2=0x7f2f080a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:54.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.150+0000 7f2f0e407700 1 --2- 172.21.15.94:0/2194757847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 0x7f2f080a0dd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009910 tx=0x7f2ef80091a0 comp rx=0 tx=0).stop 2023-09-30T00:19:54.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.151+0000 7f2f0e407700 1 -- 172.21.15.94:0/2194757847 shutdown_connections 2023-09-30T00:19:54.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.151+0000 7f2f0e407700 1 --2- 172.21.15.94:0/2194757847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 0x7f2f080a0dd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:54.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.151+0000 7f2f0e407700 1 -- 172.21.15.94:0/2194757847 >> 172.21.15.94:0/2194757847 conn(0x7f2f0806a090 msgr2=0x7f2f080702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:54.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.151+0000 7f2f0e407700 1 -- 172.21.15.94:0/2194757847 shutdown_connections 2023-09-30T00:19:54.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.151+0000 7f2f0e407700 1 -- 172.21.15.94:0/2194757847 wait complete. 2023-09-30T00:19:54.155 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.151+0000 7f2f0e407700 1 Processor -- start 2023-09-30T00:19:54.155 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.152+0000 7f2f0e407700 1 -- start start 2023-09-30T00:19:54.155 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.152+0000 7f2f0e407700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f08070c50 0x7f2f080add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:54.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.152+0000 7f2f0e407700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 0x7f2f08150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:54.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.152+0000 7f2f0e407700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f0813fd70 0x7f2f08151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:54.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.152+0000 7f2f0e407700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f08143100 con 0x7f2f08070c50 2023-09-30T00:19:54.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.152+0000 7f2f0e407700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f081508f0 con 0x7f2f0813fd70 2023-09-30T00:19:54.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.152+0000 7f2f0e407700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f081517c0 con 0x7f2f080b7160 2023-09-30T00:19:54.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.152+0000 7f2f077fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 0x7f2f08150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:54.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.152+0000 7f2f07fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f08070c50 0x7f2f080add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:54.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.152+0000 7f2f077fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 0x7f2f08150d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:45288/0 (socket says 172.21.15.94:45288) 2023-09-30T00:19:54.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.152+0000 7f2f077fe700 1 -- 172.21.15.94:0/1693146141 learned_addr learned my addr 172.21.15.94:0/1693146141 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:54.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.153+0000 7f2f0c9a4700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f0813fd70 0x7f2f08151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:54.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.153+0000 7f2f077fe700 1 -- 172.21.15.94:0/1693146141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f0813fd70 msgr2=0x7f2f08151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:54.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.153+0000 7f2f077fe700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f0813fd70 0x7f2f08151280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:54.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.153+0000 7f2f077fe700 1 -- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f08070c50 msgr2=0x7f2f080add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:54.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.153+0000 7f2f077fe700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f08070c50 0x7f2f080add70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:54.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.153+0000 7f2f077fe700 1 -- 172.21.15.94:0/1693146141 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef8009420 con 0x7f2f080b7160 2023-09-30T00:19:54.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.153+0000 7f2f077fe700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 0x7f2f08150d40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2ef80091e0 tx=0x7f2ef8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:54.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.153+0000 7f2eeffff700 1 -- 172.21.15.94:0/1693146141 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8011e10 con 0x7f2f080b7160 2023-09-30T00:19:54.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.153+0000 7f2eeffff700 1 -- 172.21.15.94:0/1693146141 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef8020070 con 0x7f2f080b7160 2023-09-30T00:19:54.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.153+0000 7f2eeffff700 1 -- 172.21.15.94:0/1693146141 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef801b6a0 con 0x7f2f080b7160 2023-09-30T00:19:54.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.153+0000 7f2f0e407700 1 -- 172.21.15.94:0/1693146141 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f080ad600 con 0x7f2f080b7160 2023-09-30T00:19:54.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.153+0000 7f2f0e407700 1 -- 172.21.15.94:0/1693146141 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f081432f0 con 0x7f2f080b7160 2023-09-30T00:19:54.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.154+0000 7f2f0e407700 1 -- 172.21.15.94:0/1693146141 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f08150a60 con 0x7f2f080b7160 2023-09-30T00:19:54.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.155+0000 7f2eeffff700 1 -- 172.21.15.94:0/1693146141 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2ef8022030 con 0x7f2f080b7160 2023-09-30T00:19:54.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.155+0000 7f2eeffff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:54.158 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.155+0000 7f2eeffff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:54.158 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.155+0000 7f2eeffff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:54.158 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.155+0000 7f2eeffff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:54.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.155+0000 7f2eeffff700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2ef4098780 0x7f2ef409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:54.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.156+0000 7f2f07fff700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2ef4098780 0x7f2ef409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:54.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.156+0000 7f2eeffff700 1 -- 172.21.15.94:0/1693146141 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f2ef80c2300 con 0x7f2f080b7160 2023-09-30T00:19:54.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.156+0000 7f2f07fff700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2ef4098780 0x7f2ef409ac30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2efc003a30 tx=0x7f2efc007ea0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:54.159 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.159+0000 7f2eeffff700 1 -- 172.21.15.94:0/1693146141 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2ef808f930 con 0x7f2f080b7160 2023-09-30T00:19:54.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.382+0000 7f2f0e407700 1 -- 172.21.15.94:0/1693146141 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2f08150a60 con 0x7f2f080b7160 2023-09-30T00:19:54.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.383+0000 7f2eeffff700 1 -- 172.21.15.94:0/1693146141 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2f08150a60 con 0x7f2f080b7160 2023-09-30T00:19:54.384 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:54.384 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:54.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 10 mgrc shutdown 2023-09-30T00:19:54.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 1 -- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2ef4098780 msgr2=0x7f2ef409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:54.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2ef4098780 0x7f2ef409ac30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2efc003a30 tx=0x7f2efc007ea0 comp rx=0 tx=0).stop 2023-09-30T00:19:54.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 1 -- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 msgr2=0x7f2f08150d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:54.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 0x7f2f08150d40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2ef80091e0 tx=0x7f2ef8016000 comp rx=0 tx=0).stop 2023-09-30T00:19:54.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 1 -- 172.21.15.94:0/1693146141 shutdown_connections 2023-09-30T00:19:54.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2ef4098780 0x7f2ef409ac30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:54.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f0813fd70 0x7f2f08151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:54.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f08070c50 0x7f2f080add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:54.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 1 --2- 172.21.15.94:0/1693146141 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2f080b7160 0x7f2f08150d40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:54.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 1 -- 172.21.15.94:0/1693146141 >> 172.21.15.94:0/1693146141 conn(0x7f2f0806a090 msgr2=0x7f2f0813ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:54.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 1 -- 172.21.15.94:0/1693146141 shutdown_connections 2023-09-30T00:19:54.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:54.385+0000 7f2f0e407700 1 -- 172.21.15.94:0/1693146141 wait complete. 2023-09-30T00:19:57.398 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:19:57.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.520+0000 7ff806f89700 1 Processor -- start 2023-09-30T00:19:57.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.521+0000 7ff806f89700 1 -- start start 2023-09-30T00:19:57.523 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.521+0000 7ff806f89700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff8000a6fc0 0x7ff800149e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:57.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.521+0000 7ff806f89700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8000a6050 con 0x7ff8000a4840 2023-09-30T00:19:57.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.521+0000 7ff806f89700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8000b4dd0 con 0x7ff8000a4470 2023-09-30T00:19:57.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.521+0000 7ff806f89700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8000a5e30 con 0x7ff8000a6fc0 2023-09-30T00:19:57.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.521+0000 7ff7fffff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff8000a4840 0x7ff8000a5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39148/0 (socket says 172.21.15.94:39148) 2023-09-30T00:19:57.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.521+0000 7ff7fffff700 1 -- 172.21.15.94:0/3318402320 learned_addr learned my addr 172.21.15.94:0/3318402320 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:19:57.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.521+0000 7ff805526700 1 --2- 172.21.15.94:0/3318402320 >> v2:172.21.15.121:3300/0 conn(0x7ff8000a6fc0 0x7ff800149e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:57.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.522+0000 7ff7ff7fe700 1 -- 172.21.15.94:0/3318402320 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459674124 0 0) 0x7ff8000a6050 con 0x7ff8000a4840 2023-09-30T00:19:57.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.522+0000 7ff7ff7fe700 1 -- 172.21.15.94:0/3318402320 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e8003570 con 0x7ff8000a4840 2023-09-30T00:19:57.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.522+0000 7ff7ff7fe700 1 -- 172.21.15.94:0/3318402320 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 912762661 0 0) 0x7ff7e8003570 con 0x7ff8000a4840 2023-09-30T00:19:57.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.522+0000 7ff805526700 1 -- 172.21.15.94:0/3318402320 >> v1:172.21.15.121:6789/0 conn(0x7ff8000a4470 legacy=0x7ff8000b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.522+0000 7ff805526700 1 -- 172.21.15.94:0/3318402320 >> v1:172.21.15.94:6789/0 conn(0x7ff8000a4840 legacy=0x7ff8000a5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.522+0000 7ff805526700 1 -- 172.21.15.94:0/3318402320 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80006ea50 con 0x7ff8000a6fc0 2023-09-30T00:19:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.522+0000 7ff805526700 1 --2- 172.21.15.94:0/3318402320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8000a6fc0 0x7ff800149e40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80098e0 tx=0x7ff7f80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36a7bff1a3b6cbab server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.522+0000 7ff7ff7fe700 1 -- 172.21.15.94:0/3318402320 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4001470 con 0x7ff8000a6fc0 2023-09-30T00:19:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.522+0000 7ff7ff7fe700 1 -- 172.21.15.94:0/3318402320 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f8011420 con 0x7ff8000a6fc0 2023-09-30T00:19:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.522+0000 7ff7ff7fe700 1 -- 172.21.15.94:0/3318402320 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f80117c0 con 0x7ff8000a6fc0 2023-09-30T00:19:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.523+0000 7ff806f89700 1 -- 172.21.15.94:0/3318402320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8000a6fc0 msgr2=0x7ff800149e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.523+0000 7ff806f89700 1 --2- 172.21.15.94:0/3318402320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8000a6fc0 0x7ff800149e40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80098e0 tx=0x7ff7f80091a0 comp rx=0 tx=0).stop 2023-09-30T00:19:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.523+0000 7ff806f89700 1 -- 172.21.15.94:0/3318402320 shutdown_connections 2023-09-30T00:19:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.523+0000 7ff806f89700 1 --2- 172.21.15.94:0/3318402320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8000a6fc0 0x7ff800149e40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.523+0000 7ff806f89700 1 -- 172.21.15.94:0/3318402320 >> 172.21.15.94:0/3318402320 conn(0x7ff80006cbf0 msgr2=0x7ff80006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.523+0000 7ff806f89700 1 -- 172.21.15.94:0/3318402320 shutdown_connections 2023-09-30T00:19:57.526 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.523+0000 7ff806f89700 1 -- 172.21.15.94:0/3318402320 wait complete. 2023-09-30T00:19:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.524+0000 7ff806f89700 1 Processor -- start 2023-09-30T00:19:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.524+0000 7ff806f89700 1 -- start start 2023-09-30T00:19:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.524+0000 7ff806f89700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8000a4470 0x7ff8001415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.524+0000 7ff806f89700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8000a4840 0x7ff8000aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.524+0000 7ff806f89700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8000a6fc0 0x7ff8001503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.524+0000 7ff806f89700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80014d810 con 0x7ff8000a6fc0 2023-09-30T00:19:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.524+0000 7ff806f89700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8000ae810 con 0x7ff8000a4840 2023-09-30T00:19:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.524+0000 7ff806f89700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8000af750 con 0x7ff8000a4470 2023-09-30T00:19:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.524+0000 7ff804d25700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8000a4470 0x7ff8001415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.524+0000 7ff805526700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8000a6fc0 0x7ff8001503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff804d25700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8000a4470 0x7ff8001415d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:47124/0 (socket says 172.21.15.94:47124) 2023-09-30T00:19:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff804d25700 1 -- 172.21.15.94:0/2478033929 learned_addr learned my addr 172.21.15.94:0/2478033929 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:19:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff805526700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8000a6fc0 0x7ff8001503b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39090/0 (socket says 172.21.15.94:39090) 2023-09-30T00:19:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff7fffff700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8000a4840 0x7ff8000aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff804d25700 1 -- 172.21.15.94:0/2478033929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8000a4840 msgr2=0x7ff8000aed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff804d25700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8000a4840 0x7ff8000aed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff804d25700 1 -- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8000a6fc0 msgr2=0x7ff8001503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff804d25700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8000a6fc0 0x7ff8001503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff804d25700 1 -- 172.21.15.94:0/2478033929 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f8009420 con 0x7ff8000a4470 2023-09-30T00:19:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff805526700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8000a6fc0 0x7ff8001503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:19:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff804d25700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8000a4470 0x7ff8001415d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff7f000c8e0 tx=0x7ff7f0001710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff7fcff9700 1 -- 172.21.15.94:0/2478033929 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0012630 con 0x7ff8000a4470 2023-09-30T00:19:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff7fcff9700 1 -- 172.21.15.94:0/2478033929 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f00127b0 con 0x7ff8000a4470 2023-09-30T00:19:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff806f89700 1 -- 172.21.15.94:0/2478033929 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8000ada70 con 0x7ff8000a4470 2023-09-30T00:19:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff7fcff9700 1 -- 172.21.15.94:0/2478033929 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0012b70 con 0x7ff8000a4470 2023-09-30T00:19:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.525+0000 7ff806f89700 1 -- 172.21.15.94:0/2478033929 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8000ad710 con 0x7ff8000a4470 2023-09-30T00:19:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.526+0000 7ff806f89700 1 -- 172.21.15.94:0/2478033929 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80006fbd0 con 0x7ff8000a4470 2023-09-30T00:19:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.527+0000 7ff7fcff9700 1 -- 172.21.15.94:0/2478033929 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ff7f0003030 con 0x7ff8000a4470 2023-09-30T00:19:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.527+0000 7ff7fcff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:19:57.531 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.527+0000 7ff7fcff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:19:57.531 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.527+0000 7ff7fcff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:57.531 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.527+0000 7ff7fcff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:19:57.531 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.527+0000 7ff7fcff9700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff7ec098780 0x7ff7ec09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:19:57.531 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.528+0000 7ff7fcff9700 1 -- 172.21.15.94:0/2478033929 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7ff7f00c0470 con 0x7ff8000a4470 2023-09-30T00:19:57.531 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.528+0000 7ff7fffff700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff7ec098780 0x7ff7ec09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:19:57.531 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.528+0000 7ff7fffff700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff7ec098780 0x7ff7ec09ac30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff800141320 tx=0x7ff7f4000bc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:19:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.531+0000 7ff7fcff9700 1 -- 172.21.15.94:0/2478033929 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff7f00915d0 con 0x7ff8000a4470 2023-09-30T00:19:57.752 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.751+0000 7ff806f89700 1 -- 172.21.15.94:0/2478033929 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff80006a720 con 0x7ff8000a4470 2023-09-30T00:19:57.752 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.752+0000 7ff7fcff9700 1 -- 172.21.15.94:0/2478033929 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff7f0091160 con 0x7ff8000a4470 2023-09-30T00:19:57.752 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:19:57.752 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:19:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 10 mgrc shutdown 2023-09-30T00:19:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 1 -- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff7ec098780 msgr2=0x7ff7ec09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff7ec098780 0x7ff7ec09ac30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff800141320 tx=0x7ff7f4000bc0 comp rx=0 tx=0).stop 2023-09-30T00:19:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 1 -- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8000a4470 msgr2=0x7ff8001415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:19:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8000a4470 0x7ff8001415d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff7f000c8e0 tx=0x7ff7f0001710 comp rx=0 tx=0).stop 2023-09-30T00:19:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 1 -- 172.21.15.94:0/2478033929 shutdown_connections 2023-09-30T00:19:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff7ec098780 0x7ff7ec09ac30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8000a4470 0x7ff8001415d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:57.756 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8000a6fc0 0x7ff8001503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:57.756 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 1 --2- 172.21.15.94:0/2478033929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8000a4840 0x7ff8000aed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:19:57.756 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 1 -- 172.21.15.94:0/2478033929 >> 172.21.15.94:0/2478033929 conn(0x7ff80006cbf0 msgr2=0x7ff8000b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:19:57.756 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 1 -- 172.21.15.94:0/2478033929 shutdown_connections 2023-09-30T00:19:57.756 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:19:57.754+0000 7ff806f89700 1 -- 172.21.15.94:0/2478033929 wait complete. 2023-09-30T00:19:59.999 INFO:tasks.ceph.mon.a.smithi094.stderr:2023-09-30T00:19:59.999+0000 7fbf72d1e700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:20:00.767 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:00.895 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.892+0000 7f8c8414b700 1 Processor -- start 2023-09-30T00:20:00.895 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.892+0000 7f8c8414b700 1 -- start start 2023-09-30T00:20:00.895 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.892+0000 7f8c8414b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8c7c0b7160 0x7f8c7c0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:00.895 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.892+0000 7f8c8414b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c13fd70 0x7f8c7c1401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:00.895 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.892+0000 7f8c8414b700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c0a1310 con 0x7f8c7c070c50 2023-09-30T00:20:00.895 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.892+0000 7f8c8414b700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c071020 con 0x7f8c7c0b7160 2023-09-30T00:20:00.895 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.892+0000 7f8c8414b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0b76a0 con 0x7f8c7c13fd70 2023-09-30T00:20:00.895 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.893+0000 7f8c826e8700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c13fd70 0x7f8c7c1401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:00.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.893+0000 7f8c826e8700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c13fd70 0x7f8c7c1401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:47134/0 (socket says 172.21.15.94:47134) 2023-09-30T00:20:00.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.893+0000 7f8c826e8700 1 -- 172.21.15.94:0/1325528704 learned_addr learned my addr 172.21.15.94:0/1325528704 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:00.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.893+0000 7f8c816e6700 1 --2- 172.21.15.94:0/1325528704 >> v2:172.21.15.121:3300/0 conn(0x7f8c7c0b7160 0x7f8c7c0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:00.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.893+0000 7f8c826e8700 1 -- 172.21.15.94:0/1325528704 >> v2:172.21.15.121:3300/0 conn(0x7f8c7c0b7160 msgr2=0x7f8c7c0a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:00.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.893+0000 7f8c826e8700 1 --2- 172.21.15.94:0/1325528704 >> v2:172.21.15.121:3300/0 conn(0x7f8c7c0b7160 0x7f8c7c0a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:00.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.893+0000 7f8c826e8700 1 -- 172.21.15.94:0/1325528704 >> v1:172.21.15.121:6789/0 conn(0x7f8c7c070c50 legacy=0x7f8c7c0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:00.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.893+0000 7f8c826e8700 1 -- 172.21.15.94:0/1325528704 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0b7940 con 0x7f8c7c13fd70 2023-09-30T00:20:00.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.893+0000 7f8c826e8700 1 --2- 172.21.15.94:0/1325528704 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c13fd70 0x7f8c7c1401d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8c780099b0 tx=0x7f8c78009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94f7954b40f92eac server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:00.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.893+0000 7f8c80ee5700 1 -- 172.21.15.94:0/1325528704 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c78009ea0 con 0x7f8c7c13fd70 2023-09-30T00:20:00.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.893+0000 7f8c80ee5700 1 -- 172.21.15.94:0/1325528704 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7800b8f0 con 0x7f8c7c13fd70 2023-09-30T00:20:00.896 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.893+0000 7f8c80ee5700 1 -- 172.21.15.94:0/1325528704 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7800bc90 con 0x7f8c7c13fd70 2023-09-30T00:20:00.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.894+0000 7f8c8414b700 1 -- 172.21.15.94:0/1325528704 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c13fd70 msgr2=0x7f8c7c1401d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:00.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.894+0000 7f8c8414b700 1 --2- 172.21.15.94:0/1325528704 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c13fd70 0x7f8c7c1401d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8c780099b0 tx=0x7f8c78009250 comp rx=0 tx=0).stop 2023-09-30T00:20:00.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.894+0000 7f8c8414b700 1 -- 172.21.15.94:0/1325528704 shutdown_connections 2023-09-30T00:20:00.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.894+0000 7f8c8414b700 1 --2- 172.21.15.94:0/1325528704 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c13fd70 0x7f8c7c1401d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:00.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.894+0000 7f8c8414b700 1 --2- 172.21.15.94:0/1325528704 >> v2:172.21.15.121:3300/0 conn(0x7f8c7c0b7160 0x7f8c7c0a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:00.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.894+0000 7f8c8414b700 1 -- 172.21.15.94:0/1325528704 >> 172.21.15.94:0/1325528704 conn(0x7f8c7c06a090 msgr2=0x7f8c7c0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:00.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.894+0000 7f8c8414b700 1 -- 172.21.15.94:0/1325528704 shutdown_connections 2023-09-30T00:20:00.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.894+0000 7f8c8414b700 1 -- 172.21.15.94:0/1325528704 wait complete. 2023-09-30T00:20:00.900 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.895+0000 7f8c8414b700 1 Processor -- start 2023-09-30T00:20:00.900 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.895+0000 7f8c8414b700 1 -- start start 2023-09-30T00:20:00.900 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.895+0000 7f8c8414b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c7c070c50 0x7f8c7c12d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:00.900 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.895+0000 7f8c8414b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c0b7160 0x7f8c7c13b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:00.900 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.895+0000 7f8c8414b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c7c13fd70 0x7f8c7c13b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:00.900 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c8414b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c1509c0 con 0x7f8c7c13fd70 2023-09-30T00:20:00.900 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c8414b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c13be60 con 0x7f8c7c070c50 2023-09-30T00:20:00.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c8414b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0a5120 con 0x7f8c7c0b7160 2023-09-30T00:20:00.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c816e6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c0b7160 0x7f8c7c13b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:00.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c826e8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c7c13fd70 0x7f8c7c13b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:00.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c816e6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c0b7160 0x7f8c7c13b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:47136/0 (socket says 172.21.15.94:47136) 2023-09-30T00:20:00.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c816e6700 1 -- 172.21.15.94:0/1903122172 learned_addr learned my addr 172.21.15.94:0/1903122172 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:00.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c81ee7700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c7c070c50 0x7f8c7c12d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:00.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c816e6700 1 -- 172.21.15.94:0/1903122172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c7c070c50 msgr2=0x7f8c7c12d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:00.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c816e6700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c7c070c50 0x7f8c7c12d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:00.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c816e6700 1 -- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c7c13fd70 msgr2=0x7f8c7c13b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:00.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c816e6700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c7c13fd70 0x7f8c7c13b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:00.901 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c816e6700 1 -- 172.21.15.94:0/1903122172 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c780094f0 con 0x7f8c7c0b7160 2023-09-30T00:20:00.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c826e8700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c7c13fd70 0x7f8c7c13b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:00.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c816e6700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c0b7160 0x7f8c7c13b3e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00ec80 tx=0x7f8c6c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:00.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c71ffb700 1 -- 172.21.15.94:0/1903122172 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c00c850 con 0x7f8c7c0b7160 2023-09-30T00:20:00.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c71ffb700 1 -- 172.21.15.94:0/1903122172 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c00c9b0 con 0x7f8c7c0b7160 2023-09-30T00:20:00.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c8414b700 1 -- 172.21.15.94:0/1903122172 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c0a53e0 con 0x7f8c7c0b7160 2023-09-30T00:20:00.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c71ffb700 1 -- 172.21.15.94:0/1903122172 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c00cd50 con 0x7f8c7c0b7160 2023-09-30T00:20:00.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.896+0000 7f8c8414b700 1 -- 172.21.15.94:0/1903122172 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c13a0e0 con 0x7f8c7c0b7160 2023-09-30T00:20:00.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.897+0000 7f8c8414b700 1 -- 172.21.15.94:0/1903122172 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c0702f0 con 0x7f8c7c0b7160 2023-09-30T00:20:00.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.898+0000 7f8c71ffb700 1 -- 172.21.15.94:0/1903122172 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f8c6c013680 con 0x7f8c7c0b7160 2023-09-30T00:20:00.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.898+0000 7f8c71ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:00.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.899+0000 7f8c71ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:00.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.899+0000 7f8c71ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:00.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.899+0000 7f8c71ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:00.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.899+0000 7f8c71ffb700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8c60098800 0x7f8c6009acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:00.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.899+0000 7f8c81ee7700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8c60098800 0x7f8c6009acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:00.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.899+0000 7f8c71ffb700 1 -- 172.21.15.94:0/1903122172 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f8c6c0c0ef0 con 0x7f8c7c0b7160 2023-09-30T00:20:00.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.899+0000 7f8c81ee7700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8c60098800 0x7f8c6009acb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8c74007fd0 tx=0x7f8c740016f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:00.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:00.902+0000 7f8c71ffb700 1 -- 172.21.15.94:0/1903122172 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8c6c08e520 con 0x7f8c7c0b7160 2023-09-30T00:20:01.125 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.125+0000 7f8c8414b700 1 -- 172.21.15.94:0/1903122172 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8c7c071630 con 0x7f8c7c0b7160 2023-09-30T00:20:01.126 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.125+0000 7f8c71ffb700 1 -- 172.21.15.94:0/1903122172 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8c6c017100 con 0x7f8c7c0b7160 2023-09-30T00:20:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:01.126 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:01.128 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 10 mgrc shutdown 2023-09-30T00:20:01.128 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 1 -- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8c60098800 msgr2=0x7f8c6009acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:01.128 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8c60098800 0x7f8c6009acb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8c74007fd0 tx=0x7f8c740016f0 comp rx=0 tx=0).stop 2023-09-30T00:20:01.128 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 1 -- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c0b7160 msgr2=0x7f8c7c13b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:01.128 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c0b7160 0x7f8c7c13b3e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00ec80 tx=0x7f8c6c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:20:01.128 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 1 -- 172.21.15.94:0/1903122172 shutdown_connections 2023-09-30T00:20:01.128 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8c60098800 0x7f8c6009acb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c7c13fd70 0x7f8c7c13b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c7c070c50 0x7f8c7c12d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 1 --2- 172.21.15.94:0/1903122172 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c7c0b7160 0x7f8c7c13b3e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 1 -- 172.21.15.94:0/1903122172 >> 172.21.15.94:0/1903122172 conn(0x7f8c7c06a090 msgr2=0x7f8c7c13cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 1 -- 172.21.15.94:0/1903122172 shutdown_connections 2023-09-30T00:20:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:01.127+0000 7f8c8414b700 1 -- 172.21.15.94:0/1903122172 wait complete. 2023-09-30T00:20:04.140 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:04.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.265+0000 7f9deac58700 1 Processor -- start 2023-09-30T00:20:04.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.265+0000 7f9deac58700 1 -- start start 2023-09-30T00:20:04.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.265+0000 7f9deac58700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9de40a0dd0 0x7f9de414c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:04.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.265+0000 7f9deac58700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de4071020 con 0x7f9de40b7160 2023-09-30T00:20:04.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.265+0000 7f9deac58700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de40b76a0 con 0x7f9de4070c50 2023-09-30T00:20:04.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.265+0000 7f9deac58700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de40b7940 con 0x7f9de40a0dd0 2023-09-30T00:20:04.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.265+0000 7f9ddbfff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9de40b7160 0x7f9de413fd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39152/0 (socket says 172.21.15.94:39152) 2023-09-30T00:20:04.267 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.265+0000 7f9ddbfff700 1 -- 172.21.15.94:0/1436589252 learned_addr learned my addr 172.21.15.94:0/1436589252 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:20:04.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.266+0000 7f9de91f5700 1 --2- 172.21.15.94:0/1436589252 >> v2:172.21.15.121:3300/0 conn(0x7f9de40a0dd0 0x7f9de414c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:04.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.266+0000 7f9ddb7fe700 1 -- 172.21.15.94:0/1436589252 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053620842 0 0) 0x7f9de4071020 con 0x7f9de40b7160 2023-09-30T00:20:04.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.266+0000 7f9ddb7fe700 1 -- 172.21.15.94:0/1436589252 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dcc003570 con 0x7f9de40b7160 2023-09-30T00:20:04.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.266+0000 7f9ddb7fe700 1 -- 172.21.15.94:0/1436589252 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3009449562 0 0) 0x7f9dcc003570 con 0x7f9de40b7160 2023-09-30T00:20:04.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.266+0000 7f9de91f5700 1 -- 172.21.15.94:0/1436589252 >> v1:172.21.15.121:6789/0 conn(0x7f9de4070c50 legacy=0x7f9de40a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:04.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.266+0000 7f9de91f5700 1 -- 172.21.15.94:0/1436589252 >> v1:172.21.15.94:6789/0 conn(0x7f9de40b7160 legacy=0x7f9de413fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:04.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.266+0000 7f9de91f5700 1 -- 172.21.15.94:0/1436589252 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de40a2070 con 0x7f9de40a0dd0 2023-09-30T00:20:04.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.266+0000 7f9de91f5700 1 --2- 172.21.15.94:0/1436589252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de40a0dd0 0x7f9de414c0d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0098e0 tx=0x7f9ddc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=190f36558733d092 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:04.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.266+0000 7f9ddb7fe700 1 -- 172.21.15.94:0/1436589252 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4001470 con 0x7f9de40a0dd0 2023-09-30T00:20:04.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.266+0000 7f9ddb7fe700 1 -- 172.21.15.94:0/1436589252 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ddc011420 con 0x7f9de40a0dd0 2023-09-30T00:20:04.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.266+0000 7f9ddb7fe700 1 -- 172.21.15.94:0/1436589252 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc0117c0 con 0x7f9de40a0dd0 2023-09-30T00:20:04.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.267+0000 7f9deac58700 1 -- 172.21.15.94:0/1436589252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de40a0dd0 msgr2=0x7f9de414c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:04.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.267+0000 7f9deac58700 1 --2- 172.21.15.94:0/1436589252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de40a0dd0 0x7f9de414c0d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0098e0 tx=0x7f9ddc0091a0 comp rx=0 tx=0).stop 2023-09-30T00:20:04.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.267+0000 7f9deac58700 1 -- 172.21.15.94:0/1436589252 shutdown_connections 2023-09-30T00:20:04.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.267+0000 7f9deac58700 1 --2- 172.21.15.94:0/1436589252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de40a0dd0 0x7f9de414c0d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:04.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.267+0000 7f9deac58700 1 -- 172.21.15.94:0/1436589252 >> 172.21.15.94:0/1436589252 conn(0x7f9de406a090 msgr2=0x7f9de40702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:04.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.267+0000 7f9deac58700 1 -- 172.21.15.94:0/1436589252 shutdown_connections 2023-09-30T00:20:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.267+0000 7f9deac58700 1 -- 172.21.15.94:0/1436589252 wait complete. 2023-09-30T00:20:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.268+0000 7f9deac58700 1 Processor -- start 2023-09-30T00:20:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.268+0000 7f9deac58700 1 -- start start 2023-09-30T00:20:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9deac58700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de4070c50 0x7f9de4143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9deac58700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9de40a0dd0 0x7f9de40a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9deac58700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de40b7160 0x7f9de40a5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9deac58700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de40a44b0 con 0x7f9de4070c50 2023-09-30T00:20:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9deac58700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de40a4770 con 0x7f9de40b7160 2023-09-30T00:20:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9deac58700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de40a4180 con 0x7f9de40a0dd0 2023-09-30T00:20:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9de89f4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de4070c50 0x7f9de4143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9ddbfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9de40a0dd0 0x7f9de40a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:04.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9de89f4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de4070c50 0x7f9de4143ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39106/0 (socket says 172.21.15.94:39106) 2023-09-30T00:20:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9de89f4700 1 -- 172.21.15.94:0/757129337 learned_addr learned my addr 172.21.15.94:0/757129337 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9de91f5700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de40b7160 0x7f9de40a5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9de89f4700 1 -- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9de40a0dd0 msgr2=0x7f9de40a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9de89f4700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9de40a0dd0 0x7f9de40a3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9de89f4700 1 -- 172.21.15.94:0/757129337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de40b7160 msgr2=0x7f9de40a5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9de89f4700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de40b7160 0x7f9de40a5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9de89f4700 1 -- 172.21.15.94:0/757129337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc009420 con 0x7f9de4070c50 2023-09-30T00:20:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9de89f4700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de4070c50 0x7f9de4143ae0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9de000cb40 tx=0x7f9de0001840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9dd8ff9700 1 -- 172.21.15.94:0/757129337 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de000ce50 con 0x7f9de4070c50 2023-09-30T00:20:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9dd8ff9700 1 -- 172.21.15.94:0/757129337 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de000a7e0 con 0x7f9de4070c50 2023-09-30T00:20:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9deac58700 1 -- 172.21.15.94:0/757129337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de4146a00 con 0x7f9de4070c50 2023-09-30T00:20:04.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9dd8ff9700 1 -- 172.21.15.94:0/757129337 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de000aba0 con 0x7f9de4070c50 2023-09-30T00:20:04.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.269+0000 7f9deac58700 1 -- 172.21.15.94:0/757129337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de4144610 con 0x7f9de4070c50 2023-09-30T00:20:04.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.270+0000 7f9deac58700 1 -- 172.21.15.94:0/757129337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de4071630 con 0x7f9de4070c50 2023-09-30T00:20:04.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.271+0000 7f9dd8ff9700 1 -- 172.21.15.94:0/757129337 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f9de0003030 con 0x7f9de4070c50 2023-09-30T00:20:04.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.271+0000 7f9dd8ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:04.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.272+0000 7f9dd8ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:04.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.272+0000 7f9dd8ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:04.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.272+0000 7f9dd8ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:04.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.272+0000 7f9dd8ff9700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9dd0098780 0x7f9dd009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:04.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.272+0000 7f9ddbfff700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9dd0098780 0x7f9dd009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:04.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.272+0000 7f9dd8ff9700 1 -- 172.21.15.94:0/757129337 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f9de00c0550 con 0x7f9de4070c50 2023-09-30T00:20:04.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.272+0000 7f9ddbfff700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9dd0098780 0x7f9dd009ac30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9de41517d0 tx=0x7f9dd4000bc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:04.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.276+0000 7f9dd8ff9700 1 -- 172.21.15.94:0/757129337 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9de008db80 con 0x7f9de4070c50 2023-09-30T00:20:04.503 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.503+0000 7f9deac58700 1 -- 172.21.15.94:0/757129337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9de413ef20 con 0x7f9de4070c50 2023-09-30T00:20:04.504 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.503+0000 7f9dd8ff9700 1 -- 172.21.15.94:0/757129337 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9de00911a0 con 0x7f9de4070c50 2023-09-30T00:20:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:04.504 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:04.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 10 mgrc shutdown 2023-09-30T00:20:04.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 1 -- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9dd0098780 msgr2=0x7f9dd009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:04.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9dd0098780 0x7f9dd009ac30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9de41517d0 tx=0x7f9dd4000bc0 comp rx=0 tx=0).stop 2023-09-30T00:20:04.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 1 -- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de4070c50 msgr2=0x7f9de4143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:04.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de4070c50 0x7f9de4143ae0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9de000cb40 tx=0x7f9de0001840 comp rx=0 tx=0).stop 2023-09-30T00:20:04.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 1 -- 172.21.15.94:0/757129337 shutdown_connections 2023-09-30T00:20:04.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f9dd0098780 0x7f9dd009ac30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:04.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de40b7160 0x7f9de40a5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:04.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de4070c50 0x7f9de4143ae0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:04.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 1 --2- 172.21.15.94:0/757129337 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9de40a0dd0 0x7f9de40a3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:04.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 1 -- 172.21.15.94:0/757129337 >> 172.21.15.94:0/757129337 conn(0x7f9de406a090 msgr2=0x7f9de413fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:04.507 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 1 -- 172.21.15.94:0/757129337 shutdown_connections 2023-09-30T00:20:04.507 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:04.505+0000 7f9deac58700 1 -- 172.21.15.94:0/757129337 wait complete. 2023-09-30T00:20:07.518 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.034+0000 7f3cc4b9e700 1 Processor -- start 2023-09-30T00:20:08.035 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.034+0000 7f3cc4b9e700 1 -- start start 2023-09-30T00:20:08.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cc4b9e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc009e8d0 0x7f3cc00b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:08.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cc4b9e700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3cc00b2470 0x7f3cc00a0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:08.037 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cc4b9e700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc013e1c0 con 0x7f3cc00a0790 2023-09-30T00:20:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cc4b9e700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cc009ecd0 con 0x7f3cc00b2470 2023-09-30T00:20:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cc4b9e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc00b1820 con 0x7f3cc009e8d0 2023-09-30T00:20:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cbf7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc009e8d0 0x7f3cc00b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cb6ffd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3cc00b2470 0x7f3cc00a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cbf7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc009e8d0 0x7f3cc00b1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44768/0 (socket says 172.21.15.94:44768) 2023-09-30T00:20:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cbf7fe700 1 -- 172.21.15.94:0/3527991136 learned_addr learned my addr 172.21.15.94:0/3527991136 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cb6ffd700 1 -- 172.21.15.94:0/3527991136 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc009e8d0 msgr2=0x7f3cc00b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cb6ffd700 1 --2- 172.21.15.94:0/3527991136 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc009e8d0 0x7f3cc00b1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cb6ffd700 1 -- 172.21.15.94:0/3527991136 >> v1:172.21.15.121:6789/0 conn(0x7f3cc00a0790 legacy=0x7f3cc006e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cb6ffd700 1 -- 172.21.15.94:0/3527991136 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc00b1ae0 con 0x7f3cc00b2470 2023-09-30T00:20:08.038 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.035+0000 7f3cbf7fe700 1 --2- 172.21.15.94:0/3527991136 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc009e8d0 0x7f3cc00b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:08.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.036+0000 7f3cb6ffd700 1 --2- 172.21.15.94:0/3527991136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cc00b2470 0x7f3cc00a0250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80098e0 tx=0x7f3ca80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c556e26c811e61e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:08.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.036+0000 7f3cbeffd700 1 -- 172.21.15.94:0/3527991136 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca800b6d0 con 0x7f3cc00b2470 2023-09-30T00:20:08.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.036+0000 7f3cbeffd700 1 -- 172.21.15.94:0/3527991136 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca800b830 con 0x7f3cc00b2470 2023-09-30T00:20:08.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.036+0000 7f3cbeffd700 1 -- 172.21.15.94:0/3527991136 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca800bbd0 con 0x7f3cc00b2470 2023-09-30T00:20:08.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.036+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3527991136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cc00b2470 msgr2=0x7f3cc00a0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:08.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.036+0000 7f3cc4b9e700 1 --2- 172.21.15.94:0/3527991136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cc00b2470 0x7f3cc00a0250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80098e0 tx=0x7f3ca80091a0 comp rx=0 tx=0).stop 2023-09-30T00:20:08.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.036+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3527991136 shutdown_connections 2023-09-30T00:20:08.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.036+0000 7f3cc4b9e700 1 --2- 172.21.15.94:0/3527991136 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc009e8d0 0x7f3cc00b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:08.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.036+0000 7f3cc4b9e700 1 --2- 172.21.15.94:0/3527991136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cc00b2470 0x7f3cc00a0250 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:08.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.036+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3527991136 >> 172.21.15.94:0/3527991136 conn(0x7f3cc006ad20 msgr2=0x7f3cc00b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:08.039 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.036+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3527991136 shutdown_connections 2023-09-30T00:20:08.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.036+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3527991136 wait complete. 2023-09-30T00:20:08.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.037+0000 7f3cc4b9e700 1 Processor -- start 2023-09-30T00:20:08.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.037+0000 7f3cc4b9e700 1 -- start start 2023-09-30T00:20:08.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cc4b9e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cc009e8d0 0x7f3cc009d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:08.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cc4b9e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc00a0790 0x7f3cc013a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:08.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cc4b9e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cc00b2470 0x7f3cc013ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:08.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cc4b9e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc01409b0 con 0x7f3cc009e8d0 2023-09-30T00:20:08.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cc4b9e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc013b070 con 0x7f3cc00b2470 2023-09-30T00:20:08.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cc4b9e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc013b280 con 0x7f3cc00a0790 2023-09-30T00:20:08.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cbf7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cc009e8d0 0x7f3cc009d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:08.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cb6ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc00a0790 0x7f3cc013a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:08.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cbf7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cc009e8d0 0x7f3cc009d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36700/0 (socket says 172.21.15.94:36700) 2023-09-30T00:20:08.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cb6ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc00a0790 0x7f3cc013a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44780/0 (socket says 172.21.15.94:44780) 2023-09-30T00:20:08.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cbf7fe700 1 -- 172.21.15.94:0/3665416897 learned_addr learned my addr 172.21.15.94:0/3665416897 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:08.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cbffff700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cc00b2470 0x7f3cc013ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:08.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cbf7fe700 1 -- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc00a0790 msgr2=0x7f3cc013a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:08.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cbf7fe700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc00a0790 0x7f3cc013a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:08.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cbf7fe700 1 -- 172.21.15.94:0/3665416897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cc00b2470 msgr2=0x7f3cc013ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:08.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cbf7fe700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cc00b2470 0x7f3cc013ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:08.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cbf7fe700 1 -- 172.21.15.94:0/3665416897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb00094f0 con 0x7f3cc009e8d0 2023-09-30T00:20:08.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cb6ffd700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc00a0790 0x7f3cc013a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:08.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.038+0000 7f3cbf7fe700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cc009e8d0 0x7f3cc009d730 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000ec80 tx=0x7f3cb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:08.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.039+0000 7f3cb7fff700 1 -- 172.21.15.94:0/3665416897 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb000c850 con 0x7f3cc009e8d0 2023-09-30T00:20:08.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.039+0000 7f3cb7fff700 1 -- 172.21.15.94:0/3665416897 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb000c9b0 con 0x7f3cc009e8d0 2023-09-30T00:20:08.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.039+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3665416897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca8009420 con 0x7f3cc009e8d0 2023-09-30T00:20:08.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.039+0000 7f3cb7fff700 1 -- 172.21.15.94:0/3665416897 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb000cd50 con 0x7f3cc009e8d0 2023-09-30T00:20:08.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.039+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3665416897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc013bcc0 con 0x7f3cc009e8d0 2023-09-30T00:20:08.044 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.040+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3665416897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc004c380 con 0x7f3cc009e8d0 2023-09-30T00:20:08.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.041+0000 7f3cb7fff700 1 -- 172.21.15.94:0/3665416897 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f3cb0013680 con 0x7f3cc009e8d0 2023-09-30T00:20:08.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.041+0000 7f3cb7fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:08.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.041+0000 7f3cb7fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:08.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.041+0000 7f3cb7fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:08.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.041+0000 7f3cb7fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:08.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.041+0000 7f3cb7fff700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3ca0098780 0x7f3ca009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:08.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.041+0000 7f3cb6ffd700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3ca0098780 0x7f3ca009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:08.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.041+0000 7f3cb7fff700 1 -- 172.21.15.94:0/3665416897 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f3cb00c0700 con 0x7f3cc009e8d0 2023-09-30T00:20:08.045 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.042+0000 7f3cb6ffd700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3ca0098780 0x7f3ca009ac30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80098e0 tx=0x7f3ca8017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:08.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.045+0000 7f3cb7fff700 1 -- 172.21.15.94:0/3665416897 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3cb008ddd0 con 0x7f3cc009e8d0 2023-09-30T00:20:08.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.269+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3665416897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3cc0069040 con 0x7f3cc009e8d0 2023-09-30T00:20:08.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.269+0000 7f3cb7fff700 1 -- 172.21.15.94:0/3665416897 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3cb00913f0 con 0x7f3cc009e8d0 2023-09-30T00:20:08.269 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:08.270 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:08.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 10 mgrc shutdown 2023-09-30T00:20:08.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3ca0098780 msgr2=0x7f3ca009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:08.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3ca0098780 0x7f3ca009ac30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80098e0 tx=0x7f3ca8017000 comp rx=0 tx=0).stop 2023-09-30T00:20:08.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cc009e8d0 msgr2=0x7f3cc009d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:08.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cc009e8d0 0x7f3cc009d730 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000ec80 tx=0x7f3cb0009250 comp rx=0 tx=0).stop 2023-09-30T00:20:08.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3665416897 shutdown_connections 2023-09-30T00:20:08.272 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3ca0098780 0x7f3ca009ac30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:08.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cc00b2470 0x7f3cc013ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:08.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cc009e8d0 0x7f3cc009d730 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:08.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 1 --2- 172.21.15.94:0/3665416897 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cc00a0790 0x7f3cc013a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:08.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3665416897 >> 172.21.15.94:0/3665416897 conn(0x7f3cc006ad20 msgr2=0x7f3cc00b0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:08.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3665416897 shutdown_connections 2023-09-30T00:20:08.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:08.271+0000 7f3cc4b9e700 1 -- 172.21.15.94:0/3665416897 wait complete. 2023-09-30T00:20:09.264 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:20:09.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:20:11.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:11.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.404+0000 7fa449f2a700 1 Processor -- start 2023-09-30T00:20:11.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.404+0000 7fa449f2a700 1 -- start start 2023-09-30T00:20:11.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa449f2a700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa4440a1080 0x7fa4440a2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:11.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa449f2a700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4440b7220 con 0x7fa44413fe20 2023-09-30T00:20:11.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa449f2a700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4440a14a0 con 0x7fa4440b6e50 2023-09-30T00:20:11.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa449f2a700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4440a36c0 con 0x7fa4440a1080 2023-09-30T00:20:11.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa449729700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa44413fe20 0x7fa44414b230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54334/0 (socket says 172.21.15.94:54334) 2023-09-30T00:20:11.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa449729700 1 -- 172.21.15.94:0/2999187996 learned_addr learned my addr 172.21.15.94:0/2999187996 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:20:11.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43bfff700 1 --2- 172.21.15.94:0/2999187996 >> v2:172.21.15.121:3300/0 conn(0x7fa4440a1080 0x7fa4440a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:11.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692456498 0 0) 0x7fa4440b7220 con 0x7fa44413fe20 2023-09-30T00:20:11.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa42c003570 con 0x7fa44413fe20 2023-09-30T00:20:11.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1086407008 0 0) 0x7fa42c003570 con 0x7fa44413fe20 2023-09-30T00:20:11.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 >> v2:172.21.15.121:3300/0 conn(0x7fa4440a1080 msgr2=0x7fa4440a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:11.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 --2- 172.21.15.94:0/2999187996 >> v2:172.21.15.121:3300/0 conn(0x7fa4440a1080 0x7fa4440a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:11.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 >> v1:172.21.15.121:6789/0 conn(0x7fa4440b6e50 legacy=0x7fa4440a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:11.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4440a2870 con 0x7fa44413fe20 2023-09-30T00:20:11.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3153706020 0 0) 0x7fa43c0017f0 con 0x7fa44413fe20 2023-09-30T00:20:11.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 >> v1:172.21.15.94:6789/0 conn(0x7fa44413fe20 legacy=0x7fa44414b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:11.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 --2- 172.21.15.94:0/2999187996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa42c003a60 0x7fa42c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:11.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 --2- 172.21.15.94:0/2999187996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa42c004810 0x7fa42c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 --2- 172.21.15.94:0/2999187996 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa42c0051c0 0x7fa42c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4440a36c0 con 0x7fa42c003a60 2023-09-30T00:20:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c00c160 con 0x7fa42c004810 2023-09-30T00:20:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.405+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c00c320 con 0x7fa42c0051c0 2023-09-30T00:20:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa448f28700 1 --2- 172.21.15.94:0/2999187996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa42c003a60 0x7fa42c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa43bfff700 1 --2- 172.21.15.94:0/2999187996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa42c004810 0x7fa42c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa449729700 1 --2- 172.21.15.94:0/2999187996 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa42c0051c0 0x7fa42c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa448f28700 1 -- 172.21.15.94:0/2999187996 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa42c0051c0 msgr2=0x7fa42c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa448f28700 1 --2- 172.21.15.94:0/2999187996 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa42c0051c0 0x7fa42c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa448f28700 1 -- 172.21.15.94:0/2999187996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa42c004810 msgr2=0x7fa42c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa448f28700 1 --2- 172.21.15.94:0/2999187996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa42c004810 0x7fa42c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa448f28700 1 -- 172.21.15.94:0/2999187996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c003570 con 0x7fa42c003a60 2023-09-30T00:20:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa449729700 1 -- 172.21.15.94:0/2999187996 reap_dead start 2023-09-30T00:20:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa448f28700 1 --2- 172.21.15.94:0/2999187996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa42c003a60 0x7fa42c003e70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa4400018b0 tx=0x7fa440007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13de42db7a75fbfb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440012420 con 0x7fa42c003a60 2023-09-30T00:20:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4400125a0 con 0x7fa42c003a60 2023-09-30T00:20:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.406+0000 7fa43b7fe700 1 -- 172.21.15.94:0/2999187996 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440012960 con 0x7fa42c003a60 2023-09-30T00:20:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.407+0000 7fa449f2a700 1 -- 172.21.15.94:0/2999187996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa42c003a60 msgr2=0x7fa42c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.407+0000 7fa449f2a700 1 --2- 172.21.15.94:0/2999187996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa42c003a60 0x7fa42c003e70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa4400018b0 tx=0x7fa440007d30 comp rx=0 tx=0).stop 2023-09-30T00:20:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.407+0000 7fa449f2a700 1 -- 172.21.15.94:0/2999187996 shutdown_connections 2023-09-30T00:20:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.407+0000 7fa449f2a700 1 --2- 172.21.15.94:0/2999187996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa42c003a60 0x7fa42c003e70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.407+0000 7fa449f2a700 1 -- 172.21.15.94:0/2999187996 >> 172.21.15.94:0/2999187996 conn(0x7fa4440702f0 msgr2=0x7fa4440706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.407+0000 7fa449f2a700 1 -- 172.21.15.94:0/2999187996 shutdown_connections 2023-09-30T00:20:11.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.407+0000 7fa449f2a700 1 -- 172.21.15.94:0/2999187996 wait complete. 2023-09-30T00:20:11.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa449f2a700 1 Processor -- start 2023-09-30T00:20:11.412 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa449f2a700 1 -- start start 2023-09-30T00:20:11.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa449f2a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa42c003a60 0x7fa4440ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:11.414 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa449f2a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4440adb30 0x7fa4441462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:11.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa449f2a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4441467f0 0x7fa444143750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:11.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa449f2a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4440acc10 con 0x7fa4441467f0 2023-09-30T00:20:11.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa449f2a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa444146c90 con 0x7fa4440adb30 2023-09-30T00:20:11.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa449f2a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4440aac60 con 0x7fa42c003a60 2023-09-30T00:20:11.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa448f28700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa42c003a60 0x7fa4440ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:11.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa449729700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4441467f0 0x7fa444143750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:11.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa448f28700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa42c003a60 0x7fa4440ad5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44804/0 (socket says 172.21.15.94:44804) 2023-09-30T00:20:11.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa449729700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4441467f0 0x7fa444143750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36732/0 (socket says 172.21.15.94:36732) 2023-09-30T00:20:11.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa448f28700 1 -- 172.21.15.94:0/4286835744 learned_addr learned my addr 172.21.15.94:0/4286835744 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:11.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.408+0000 7fa43bfff700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4440adb30 0x7fa4441462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:11.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.409+0000 7fa448f28700 1 -- 172.21.15.94:0/4286835744 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4440adb30 msgr2=0x7fa4441462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:11.415 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.409+0000 7fa448f28700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4440adb30 0x7fa4441462b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:11.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.409+0000 7fa448f28700 1 -- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4441467f0 msgr2=0x7fa444143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:11.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.409+0000 7fa448f28700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4441467f0 0x7fa444143750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:11.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.409+0000 7fa448f28700 1 -- 172.21.15.94:0/4286835744 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4440aaf20 con 0x7fa42c003a60 2023-09-30T00:20:11.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.409+0000 7fa449729700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4441467f0 0x7fa444143750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:11.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.409+0000 7fa448f28700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa42c003a60 0x7fa4440ad5f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00c130 tx=0x7fa440015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:11.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.409+0000 7fa438ff9700 1 -- 172.21.15.94:0/4286835744 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44001f6f0 con 0x7fa42c003a60 2023-09-30T00:20:11.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.409+0000 7fa438ff9700 1 -- 172.21.15.94:0/4286835744 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44001f850 con 0x7fa42c003a60 2023-09-30T00:20:11.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.409+0000 7fa449f2a700 1 -- 172.21.15.94:0/4286835744 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa444143130 con 0x7fa42c003a60 2023-09-30T00:20:11.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.409+0000 7fa438ff9700 1 -- 172.21.15.94:0/4286835744 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44001fbd0 con 0x7fa42c003a60 2023-09-30T00:20:11.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.409+0000 7fa449f2a700 1 -- 172.21.15.94:0/4286835744 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa444143c90 con 0x7fa42c003a60 2023-09-30T00:20:11.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.410+0000 7fa449f2a700 1 -- 172.21.15.94:0/4286835744 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4441435a0 con 0x7fa42c003a60 2023-09-30T00:20:11.416 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.411+0000 7fa438ff9700 1 -- 172.21.15.94:0/4286835744 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fa44000c420 con 0x7fa42c003a60 2023-09-30T00:20:11.417 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.411+0000 7fa438ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:11.417 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.411+0000 7fa438ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:11.417 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.411+0000 7fa438ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:11.417 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.411+0000 7fa438ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:11.417 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.411+0000 7fa438ff9700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa430098780 0x7fa43009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:11.417 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.411+0000 7fa43bfff700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa430098780 0x7fa43009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:11.417 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.411+0000 7fa438ff9700 1 -- 172.21.15.94:0/4286835744 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fa4400c1c40 con 0x7fa42c003a60 2023-09-30T00:20:11.417 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.412+0000 7fa43bfff700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa430098780 0x7fa43009ac30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa444141340 tx=0x7fa4340091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:11.417 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.415+0000 7fa438ff9700 1 -- 172.21.15.94:0/4286835744 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa44008f270 con 0x7fa42c003a60 2023-09-30T00:20:11.637 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.636+0000 7fa449f2a700 1 -- 172.21.15.94:0/4286835744 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa444066e50 con 0x7fa42c003a60 2023-09-30T00:20:11.637 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.636+0000 7fa438ff9700 1 -- 172.21.15.94:0/4286835744 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa440023100 con 0x7fa42c003a60 2023-09-30T00:20:11.637 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:11.637 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:11.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 10 mgrc shutdown 2023-09-30T00:20:11.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 1 -- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa430098780 msgr2=0x7fa43009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:11.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa430098780 0x7fa43009ac30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa444141340 tx=0x7fa4340091a0 comp rx=0 tx=0).stop 2023-09-30T00:20:11.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 1 -- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa42c003a60 msgr2=0x7fa4440ad5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:11.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa42c003a60 0x7fa4440ad5f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00c130 tx=0x7fa440015000 comp rx=0 tx=0).stop 2023-09-30T00:20:11.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 1 -- 172.21.15.94:0/4286835744 shutdown_connections 2023-09-30T00:20:11.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa430098780 0x7fa43009ac30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:11.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa42c003a60 0x7fa4440ad5f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:11.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4441467f0 0x7fa444143750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:11.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 1 --2- 172.21.15.94:0/4286835744 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4440adb30 0x7fa4441462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:11.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 1 -- 172.21.15.94:0/4286835744 >> 172.21.15.94:0/4286835744 conn(0x7fa4440702f0 msgr2=0x7fa44413f250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:11.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 1 -- 172.21.15.94:0/4286835744 shutdown_connections 2023-09-30T00:20:11.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:11.638+0000 7fa449f2a700 1 -- 172.21.15.94:0/4286835744 wait complete. 2023-09-30T00:20:14.651 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:14.784 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.777+0000 7fecb5586700 1 Processor -- start 2023-09-30T00:20:14.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.777+0000 7fecb5586700 1 -- start start 2023-09-30T00:20:14.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.777+0000 7fecb5586700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fecb00a4470 0x7fecb00b5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:14.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.777+0000 7fecb5586700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fecb00a4840 0x7fecb00a5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:14.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.777+0000 7fecb5586700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a6fc0 0x7fecb0141a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:14.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.777+0000 7fecb5586700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb00a6050 con 0x7fecb00a4840 2023-09-30T00:20:14.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.777+0000 7fecb5586700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb00b4dd0 con 0x7fecb00a4470 2023-09-30T00:20:14.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.777+0000 7fecb5586700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb006ea50 con 0x7fecb00a6fc0 2023-09-30T00:20:14.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.777+0000 7feca67fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fecb00a4840 0x7fecb00a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:14.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.777+0000 7fecaf7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a6fc0 0x7fecb0141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:14.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.777+0000 7feca67fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fecb00a4840 0x7fecb00a5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58570/0 (socket says 172.21.15.94:58570) 2023-09-30T00:20:14.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.777+0000 7feca67fc700 1 -- 172.21.15.94:0/1188890613 learned_addr learned my addr 172.21.15.94:0/1188890613 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:14.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7fecaeffd700 1 --2- 172.21.15.94:0/1188890613 >> v2:172.21.15.121:3300/0 conn(0x7fecb00a4470 0x7fecb00b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:14.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7feca67fc700 1 -- 172.21.15.94:0/1188890613 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a6fc0 msgr2=0x7fecb0141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:14.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7feca67fc700 1 --2- 172.21.15.94:0/1188890613 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a6fc0 0x7fecb0141a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:14.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7feca67fc700 1 -- 172.21.15.94:0/1188890613 >> v2:172.21.15.121:3300/0 conn(0x7fecb00a4470 msgr2=0x7fecb00b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:14.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7feca67fc700 1 --2- 172.21.15.94:0/1188890613 >> v2:172.21.15.121:3300/0 conn(0x7fecb00a4470 0x7fecb00b5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:14.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7feca67fc700 1 -- 172.21.15.94:0/1188890613 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb00b5a90 con 0x7fecb00a4840 2023-09-30T00:20:14.787 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7fecaf7fe700 1 --2- 172.21.15.94:0/1188890613 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a6fc0 0x7fecb0141a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:14.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7fecaeffd700 1 --2- 172.21.15.94:0/1188890613 >> v2:172.21.15.121:3300/0 conn(0x7fecb00a4470 0x7fecb00b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:20:14.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7feca67fc700 1 --2- 172.21.15.94:0/1188890613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fecb00a4840 0x7fecb00a5720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fecb00a4f30 tx=0x7fec98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5341052dd673654b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:14.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7fecae7fc700 1 -- 172.21.15.94:0/1188890613 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec98009e70 con 0x7fecb00a4840 2023-09-30T00:20:14.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7fecae7fc700 1 -- 172.21.15.94:0/1188890613 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9800b860 con 0x7fecb00a4840 2023-09-30T00:20:14.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7fecae7fc700 1 -- 172.21.15.94:0/1188890613 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9800bc00 con 0x7fecb00a4840 2023-09-30T00:20:14.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7fecb5586700 1 -- 172.21.15.94:0/1188890613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fecb00a4840 msgr2=0x7fecb00a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:14.788 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.778+0000 7fecb5586700 1 --2- 172.21.15.94:0/1188890613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fecb00a4840 0x7fecb00a5720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fecb00a4f30 tx=0x7fec98009250 comp rx=0 tx=0).stop 2023-09-30T00:20:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.779+0000 7fecb5586700 1 -- 172.21.15.94:0/1188890613 shutdown_connections 2023-09-30T00:20:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.779+0000 7fecb5586700 1 --2- 172.21.15.94:0/1188890613 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a6fc0 0x7fecb0141a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.779+0000 7fecb5586700 1 --2- 172.21.15.94:0/1188890613 >> v2:172.21.15.121:3300/0 conn(0x7fecb00a4470 0x7fecb00b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.779+0000 7fecb5586700 1 --2- 172.21.15.94:0/1188890613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fecb00a4840 0x7fecb00a5720 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.779+0000 7fecb5586700 1 -- 172.21.15.94:0/1188890613 >> 172.21.15.94:0/1188890613 conn(0x7fecb006cbf0 msgr2=0x7fecb006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:14.789 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.779+0000 7fecb5586700 1 -- 172.21.15.94:0/1188890613 shutdown_connections 2023-09-30T00:20:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.779+0000 7fecb5586700 1 -- 172.21.15.94:0/1188890613 wait complete. 2023-09-30T00:20:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.780+0000 7fecb5586700 1 Processor -- start 2023-09-30T00:20:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.780+0000 7fecb5586700 1 -- start start 2023-09-30T00:20:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.780+0000 7fecb5586700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb00a4470 0x7fecb013d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.780+0000 7fecb5586700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a4840 0x7fecb00b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:14.790 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.780+0000 7fecb5586700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fecb00a6fc0 0x7fecb00b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.780+0000 7fecb5586700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb006a500 con 0x7fecb00a6fc0 2023-09-30T00:20:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.780+0000 7fecb5586700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb00af9b0 con 0x7fecb00a4470 2023-09-30T00:20:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.780+0000 7fecb5586700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb0068840 con 0x7fecb00a4840 2023-09-30T00:20:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.780+0000 7feca67fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a4840 0x7fecb00b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.780+0000 7fecaf7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fecb00a6fc0 0x7fecb00b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:14.791 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.780+0000 7feca67fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a4840 0x7fecb00b06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:40322/0 (socket says 172.21.15.94:40322) 2023-09-30T00:20:14.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.780+0000 7feca67fc700 1 -- 172.21.15.94:0/240606321 learned_addr learned my addr 172.21.15.94:0/240606321 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:14.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7fecaeffd700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb00a4470 0x7fecb013d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:14.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7fecaf7fe700 1 -- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a4840 msgr2=0x7fecb00b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:14.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7fecaf7fe700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a4840 0x7fecb00b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:14.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7fecaf7fe700 1 -- 172.21.15.94:0/240606321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb00a4470 msgr2=0x7fecb013d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:14.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7fecaf7fe700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb00a4470 0x7fecb013d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:14.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7fecaf7fe700 1 -- 172.21.15.94:0/240606321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca8009a20 con 0x7fecb00a6fc0 2023-09-30T00:20:14.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7feca67fc700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a4840 0x7fecb00b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:14.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7fecaf7fe700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fecb00a6fc0 0x7fecb00b0bf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feca8009f10 tx=0x7feca800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:14.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7feca77fe700 1 -- 172.21.15.94:0/240606321 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca800cc20 con 0x7fecb00a6fc0 2023-09-30T00:20:14.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7feca77fe700 1 -- 172.21.15.94:0/240606321 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca800f040 con 0x7fecb00a6fc0 2023-09-30T00:20:14.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7fecb5586700 1 -- 172.21.15.94:0/240606321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec980094f0 con 0x7fecb00a6fc0 2023-09-30T00:20:14.793 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7feca77fe700 1 -- 172.21.15.94:0/240606321 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca8014480 con 0x7fecb00a6fc0 2023-09-30T00:20:14.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.781+0000 7fecb5586700 1 -- 172.21.15.94:0/240606321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb0069830 con 0x7fecb00a6fc0 2023-09-30T00:20:14.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.782+0000 7fecb5586700 1 -- 172.21.15.94:0/240606321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb006a6b0 con 0x7fecb00a6fc0 2023-09-30T00:20:14.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.783+0000 7feca77fe700 1 -- 172.21.15.94:0/240606321 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7feca8003020 con 0x7fecb00a6fc0 2023-09-30T00:20:14.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.783+0000 7feca77fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:14.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.784+0000 7feca77fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:14.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.784+0000 7feca77fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:14.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.784+0000 7feca77fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:14.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.784+0000 7feca77fe700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fec900987b0 0x7fec9009ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:14.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.784+0000 7fecaeffd700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fec900987b0 0x7fec9009ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:14.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.784+0000 7feca77fe700 1 -- 172.21.15.94:0/240606321 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7feca80c0d70 con 0x7fecb00a6fc0 2023-09-30T00:20:14.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.784+0000 7fecaeffd700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fec900987b0 0x7fec9009ac60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feca0006fd0 tx=0x7feca0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:14.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:14.788+0000 7feca77fe700 1 -- 172.21.15.94:0/240606321 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fecb006a6b0 con 0x7fecb00a6fc0 2023-09-30T00:20:15.012 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.011+0000 7fecb5586700 1 -- 172.21.15.94:0/240606321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fecb006a6b0 con 0x7fecb00a6fc0 2023-09-30T00:20:15.012 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.012+0000 7feca77fe700 1 -- 172.21.15.94:0/240606321 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fecb006a6b0 con 0x7fecb00a6fc0 2023-09-30T00:20:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:15.013 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:15.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 10 mgrc shutdown 2023-09-30T00:20:15.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 1 -- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fec900987b0 msgr2=0x7fec9009ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:15.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fec900987b0 0x7fec9009ac60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feca0006fd0 tx=0x7feca0008040 comp rx=0 tx=0).stop 2023-09-30T00:20:15.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 1 -- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fecb00a6fc0 msgr2=0x7fecb00b0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:15.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fecb00a6fc0 0x7fecb00b0bf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feca8009f10 tx=0x7feca800b040 comp rx=0 tx=0).stop 2023-09-30T00:20:15.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 1 -- 172.21.15.94:0/240606321 shutdown_connections 2023-09-30T00:20:15.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fec900987b0 0x7fec9009ac60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:15.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fecb00a6fc0 0x7fecb00b0bf0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:15.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb00a4470 0x7fecb013d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:15.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 1 --2- 172.21.15.94:0/240606321 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fecb00a4840 0x7fecb00b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:15.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 1 -- 172.21.15.94:0/240606321 >> 172.21.15.94:0/240606321 conn(0x7fecb006cbf0 msgr2=0x7fecb00b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:15.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 1 -- 172.21.15.94:0/240606321 shutdown_connections 2023-09-30T00:20:15.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:15.014+0000 7fecb5586700 1 -- 172.21.15.94:0/240606321 wait complete. 2023-09-30T00:20:18.027 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:18.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.152+0000 7fc44f5e6700 1 Processor -- start 2023-09-30T00:20:18.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.152+0000 7fc44f5e6700 1 -- start start 2023-09-30T00:20:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44f5e6700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc448067c60 0x7fc4480b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44f5e6700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc4480a5720 0x7fc4480a6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44f5e6700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4480a62b0 con 0x7fc448067890 2023-09-30T00:20:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44f5e6700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4480a6090 con 0x7fc448067c60 2023-09-30T00:20:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44f5e6700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4480a6570 con 0x7fc4480a5720 2023-09-30T00:20:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44cb81700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc448067c60 0x7fc4480b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:18.155 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44cb81700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc448067c60 0x7fc4480b4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58588/0 (socket says 172.21.15.94:58588) 2023-09-30T00:20:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44cb81700 1 -- 172.21.15.94:0/1487344721 learned_addr learned my addr 172.21.15.94:0/1487344721 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44db83700 1 --2- 172.21.15.94:0/1487344721 >> v2:172.21.15.121:3300/0 conn(0x7fc4480a5720 0x7fc4480a6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44cb81700 1 -- 172.21.15.94:0/1487344721 >> v2:172.21.15.121:3300/0 conn(0x7fc4480a5720 msgr2=0x7fc4480a6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44cb81700 1 --2- 172.21.15.94:0/1487344721 >> v2:172.21.15.121:3300/0 conn(0x7fc4480a5720 0x7fc4480a6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44cb81700 1 -- 172.21.15.94:0/1487344721 >> v1:172.21.15.121:6789/0 conn(0x7fc448067890 legacy=0x7fc4480b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44cb81700 1 -- 172.21.15.94:0/1487344721 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4480a6830 con 0x7fc448067c60 2023-09-30T00:20:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44db83700 1 --2- 172.21.15.94:0/1487344721 >> v2:172.21.15.121:3300/0 conn(0x7fc4480a5720 0x7fc4480a6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:20:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc44cb81700 1 --2- 172.21.15.94:0/1487344721 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc448067c60 0x7fc4480b4bd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc44806dfa0 tx=0x7fc438009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e965fa0ca6975c1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.153+0000 7fc43ffff700 1 -- 172.21.15.94:0/1487344721 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43800b820 con 0x7fc448067c60 2023-09-30T00:20:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.154+0000 7fc43ffff700 1 -- 172.21.15.94:0/1487344721 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43800b980 con 0x7fc448067c60 2023-09-30T00:20:18.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.154+0000 7fc43ffff700 1 -- 172.21.15.94:0/1487344721 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43800bd20 con 0x7fc448067c60 2023-09-30T00:20:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.154+0000 7fc44f5e6700 1 -- 172.21.15.94:0/1487344721 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc448067c60 msgr2=0x7fc4480b4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.154+0000 7fc44f5e6700 1 --2- 172.21.15.94:0/1487344721 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc448067c60 0x7fc4480b4bd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc44806dfa0 tx=0x7fc438009250 comp rx=0 tx=0).stop 2023-09-30T00:20:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.154+0000 7fc44f5e6700 1 -- 172.21.15.94:0/1487344721 shutdown_connections 2023-09-30T00:20:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.154+0000 7fc44f5e6700 1 --2- 172.21.15.94:0/1487344721 >> v2:172.21.15.121:3300/0 conn(0x7fc4480a5720 0x7fc4480a6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.154+0000 7fc44f5e6700 1 --2- 172.21.15.94:0/1487344721 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc448067c60 0x7fc4480b4bd0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.154+0000 7fc44f5e6700 1 -- 172.21.15.94:0/1487344721 >> 172.21.15.94:0/1487344721 conn(0x7fc44806cb10 msgr2=0x7fc44806cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.154+0000 7fc44f5e6700 1 -- 172.21.15.94:0/1487344721 shutdown_connections 2023-09-30T00:20:18.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.154+0000 7fc44f5e6700 1 -- 172.21.15.94:0/1487344721 wait complete. 2023-09-30T00:20:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.155+0000 7fc44f5e6700 1 Processor -- start 2023-09-30T00:20:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.155+0000 7fc44f5e6700 1 -- start start 2023-09-30T00:20:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44f5e6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc448067890 0x7fc4480a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44f5e6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc448067c60 0x7fc448141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44f5e6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4480a5720 0x7fc44809a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:18.160 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44f5e6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4481416a0 con 0x7fc448067c60 2023-09-30T00:20:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44f5e6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc448153d30 con 0x7fc448067890 2023-09-30T00:20:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44f5e6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc448153ff0 con 0x7fc4480a5720 2023-09-30T00:20:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44cb81700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc448067c60 0x7fc448141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44db83700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4480a5720 0x7fc44809a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44cb81700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc448067c60 0x7fc448141160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58602/0 (socket says 172.21.15.94:58602) 2023-09-30T00:20:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44cb81700 1 -- 172.21.15.94:0/2492733538 learned_addr learned my addr 172.21.15.94:0/2492733538 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44db83700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4480a5720 0x7fc44809a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:40336/0 (socket says 172.21.15.94:40336) 2023-09-30T00:20:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44d382700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc448067890 0x7fc4480a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44db83700 1 -- 172.21.15.94:0/2492733538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc448067890 msgr2=0x7fc4480a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44db83700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc448067890 0x7fc4480a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:18.161 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44db83700 1 -- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc448067c60 msgr2=0x7fc448141160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44db83700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc448067c60 0x7fc448141160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44db83700 1 -- 172.21.15.94:0/2492733538 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4380094f0 con 0x7fc4480a5720 2023-09-30T00:20:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44cb81700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc448067c60 0x7fc448141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44db83700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4480a5720 0x7fc44809a9c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc44000cf60 tx=0x7fc440007840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc43d7fa700 1 -- 172.21.15.94:0/2492733538 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc440015440 con 0x7fc4480a5720 2023-09-30T00:20:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc43d7fa700 1 -- 172.21.15.94:0/2492733538 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4400155a0 con 0x7fc4480a5720 2023-09-30T00:20:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc44f5e6700 1 -- 172.21.15.94:0/2492733538 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4480a1340 con 0x7fc4480a5720 2023-09-30T00:20:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.156+0000 7fc43d7fa700 1 -- 172.21.15.94:0/2492733538 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc440015960 con 0x7fc4480a5720 2023-09-30T00:20:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.157+0000 7fc44f5e6700 1 -- 172.21.15.94:0/2492733538 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc448153930 con 0x7fc4480a5720 2023-09-30T00:20:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.157+0000 7fc44f5e6700 1 -- 172.21.15.94:0/2492733538 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc448071630 con 0x7fc4480a5720 2023-09-30T00:20:18.162 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.158+0000 7fc43d7fa700 1 -- 172.21.15.94:0/2492733538 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc440015d50 con 0x7fc4480a5720 2023-09-30T00:20:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.158+0000 7fc43d7fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.159+0000 7fc43d7fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.159+0000 7fc43d7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.159+0000 7fc43d7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.159+0000 7fc43d7fa700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc42c098780 0x7fc42c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.159+0000 7fc44d382700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc42c098780 0x7fc42c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.159+0000 7fc43d7fa700 1 -- 172.21.15.94:0/2492733538 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fc4400c0d80 con 0x7fc4480a5720 2023-09-30T00:20:18.163 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.159+0000 7fc44d382700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc42c098780 0x7fc42c09ac30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc444006fd0 tx=0x7fc444008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.163+0000 7fc43d7fa700 1 -- 172.21.15.94:0/2492733538 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc44008e3c0 con 0x7fc4480a5720 2023-09-30T00:20:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.383+0000 7fc44f5e6700 1 -- 172.21.15.94:0/2492733538 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc448069f70 con 0x7fc4480a5720 2023-09-30T00:20:18.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.383+0000 7fc43d7fa700 1 -- 172.21.15.94:0/2492733538 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc44001a020 con 0x7fc4480a5720 2023-09-30T00:20:18.384 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:18.384 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:18.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 10 mgrc shutdown 2023-09-30T00:20:18.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 1 -- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc42c098780 msgr2=0x7fc42c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:18.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc42c098780 0x7fc42c09ac30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc444006fd0 tx=0x7fc444008040 comp rx=0 tx=0).stop 2023-09-30T00:20:18.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 1 -- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4480a5720 msgr2=0x7fc44809a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:18.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4480a5720 0x7fc44809a9c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc44000cf60 tx=0x7fc440007840 comp rx=0 tx=0).stop 2023-09-30T00:20:18.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 1 -- 172.21.15.94:0/2492733538 shutdown_connections 2023-09-30T00:20:18.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc42c098780 0x7fc42c09ac30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:18.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc4480a5720 0x7fc44809a9c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:18.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc448067890 0x7fc4480a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:18.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 1 --2- 172.21.15.94:0/2492733538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc448067c60 0x7fc448141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:18.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 1 -- 172.21.15.94:0/2492733538 >> 172.21.15.94:0/2492733538 conn(0x7fc44806cb10 msgr2=0x7fc448134850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:18.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 1 -- 172.21.15.94:0/2492733538 shutdown_connections 2023-09-30T00:20:18.387 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:18.385+0000 7fc44f5e6700 1 -- 172.21.15.94:0/2492733538 wait complete. 2023-09-30T00:20:21.398 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.522+0000 7ff12c7f2700 1 Processor -- start 2023-09-30T00:20:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.522+0000 7ff12c7f2700 1 -- start start 2023-09-30T00:20:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.523+0000 7ff12c7f2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 0x7ff1240a5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.523+0000 7ff12c7f2700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1240a6050 con 0x7ff1240a6fc0 2023-09-30T00:20:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.523+0000 7ff12c7f2700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1240b4dd0 con 0x7ff1240a4470 2023-09-30T00:20:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.523+0000 7ff12c7f2700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12406ea50 con 0x7ff1240a4840 2023-09-30T00:20:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.523+0000 7ff129d8d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 0x7ff1240a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:21.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.523+0000 7ff12ad8f700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff1240a6fc0 0x7ff124148e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59062/0 (socket says 172.21.15.94:59062) 2023-09-30T00:20:21.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.523+0000 7ff12ad8f700 1 -- 172.21.15.94:0/4196599134 learned_addr learned my addr 172.21.15.94:0/4196599134 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:20:21.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.523+0000 7ff129d8d700 1 -- 172.21.15.94:0/4196599134 >> v1:172.21.15.121:6789/0 conn(0x7ff1240a4470 legacy=0x7ff1240b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:21.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.523+0000 7ff12958c700 1 -- 172.21.15.94:0/4196599134 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126297235 0 0) 0x7ff1240a6050 con 0x7ff1240a6fc0 2023-09-30T00:20:21.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.523+0000 7ff129d8d700 1 -- 172.21.15.94:0/4196599134 >> v1:172.21.15.94:6789/0 conn(0x7ff1240a6fc0 legacy=0x7ff124148e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:21.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.523+0000 7ff129d8d700 1 -- 172.21.15.94:0/4196599134 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1240a5c60 con 0x7ff1240a4840 2023-09-30T00:20:21.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.524+0000 7ff129d8d700 1 --2- 172.21.15.94:0/4196599134 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 0x7ff1240a5720 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff114009910 tx=0x7ff1140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6ed87176ccde2b7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:21.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.524+0000 7ff12958c700 1 -- 172.21.15.94:0/4196599134 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11400b860 con 0x7ff1240a4840 2023-09-30T00:20:21.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.524+0000 7ff12958c700 1 -- 172.21.15.94:0/4196599134 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11400b9c0 con 0x7ff1240a4840 2023-09-30T00:20:21.525 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.524+0000 7ff12958c700 1 -- 172.21.15.94:0/4196599134 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11400bd60 con 0x7ff1240a4840 2023-09-30T00:20:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.524+0000 7ff12c7f2700 1 -- 172.21.15.94:0/4196599134 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 msgr2=0x7ff1240a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.524+0000 7ff12c7f2700 1 --2- 172.21.15.94:0/4196599134 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 0x7ff1240a5720 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff114009910 tx=0x7ff1140091a0 comp rx=0 tx=0).stop 2023-09-30T00:20:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.524+0000 7ff12c7f2700 1 -- 172.21.15.94:0/4196599134 shutdown_connections 2023-09-30T00:20:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.524+0000 7ff12c7f2700 1 --2- 172.21.15.94:0/4196599134 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 0x7ff1240a5720 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.524+0000 7ff12c7f2700 1 -- 172.21.15.94:0/4196599134 >> 172.21.15.94:0/4196599134 conn(0x7ff12406cbf0 msgr2=0x7ff12406cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.524+0000 7ff12c7f2700 1 -- 172.21.15.94:0/4196599134 shutdown_connections 2023-09-30T00:20:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.524+0000 7ff12c7f2700 1 -- 172.21.15.94:0/4196599134 wait complete. 2023-09-30T00:20:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.525+0000 7ff12c7f2700 1 Processor -- start 2023-09-30T00:20:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.525+0000 7ff12c7f2700 1 -- start start 2023-09-30T00:20:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12c7f2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1240a4470 0x7ff1240aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12c7f2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 0x7ff1240ac4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12c7f2700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1240a6fc0 0x7ff12414e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12c7f2700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240aca10 con 0x7ff1240a6fc0 2023-09-30T00:20:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12c7f2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12414ee80 con 0x7ff1240a4470 2023-09-30T00:20:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12c7f2700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240a9730 con 0x7ff1240a4840 2023-09-30T00:20:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12ad8f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1240a6fc0 0x7ff12414e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff129d8d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 0x7ff1240ac4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff129d8d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 0x7ff1240ac4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:40348/0 (socket says 172.21.15.94:40348) 2023-09-30T00:20:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12ad8f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1240a6fc0 0x7ff12414e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58618/0 (socket says 172.21.15.94:58618) 2023-09-30T00:20:21.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff129d8d700 1 -- 172.21.15.94:0/3655454307 learned_addr learned my addr 172.21.15.94:0/3655454307 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:21.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12a58e700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1240a4470 0x7ff1240aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:21.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12ad8f700 1 -- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 msgr2=0x7ff1240ac4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:21.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12ad8f700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 0x7ff1240ac4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:21.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12ad8f700 1 -- 172.21.15.94:0/3655454307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1240a4470 msgr2=0x7ff1240aaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:21.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12ad8f700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1240a4470 0x7ff1240aaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:21.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12ad8f700 1 -- 172.21.15.94:0/3655454307 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff114009420 con 0x7ff1240a6fc0 2023-09-30T00:20:21.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff129d8d700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 0x7ff1240ac4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:21.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.526+0000 7ff12ad8f700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1240a6fc0 0x7ff12414e1d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff11c001eb0 tx=0x7ff11c007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:21.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.527+0000 7ff11a7fc700 1 -- 172.21.15.94:0/3655454307 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c005b30 con 0x7ff1240a6fc0 2023-09-30T00:20:21.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.527+0000 7ff11a7fc700 1 -- 172.21.15.94:0/3655454307 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11c005cf0 con 0x7ff1240a6fc0 2023-09-30T00:20:21.531 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.527+0000 7ff12c7f2700 1 -- 172.21.15.94:0/3655454307 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff124142680 con 0x7ff1240a6fc0 2023-09-30T00:20:21.531 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.527+0000 7ff11a7fc700 1 -- 172.21.15.94:0/3655454307 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c007320 con 0x7ff1240a6fc0 2023-09-30T00:20:21.531 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.527+0000 7ff12c7f2700 1 -- 172.21.15.94:0/3655454307 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff12414e950 con 0x7ff1240a6fc0 2023-09-30T00:20:21.531 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.528+0000 7ff12c7f2700 1 -- 172.21.15.94:0/3655454307 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12406fbd0 con 0x7ff1240a6fc0 2023-09-30T00:20:21.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.529+0000 7ff11a7fc700 1 -- 172.21.15.94:0/3655454307 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ff11c01b3b0 con 0x7ff1240a6fc0 2023-09-30T00:20:21.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.529+0000 7ff11a7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:21.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.529+0000 7ff11a7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:21.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.529+0000 7ff11a7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:21.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.529+0000 7ff11a7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:21.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.529+0000 7ff11a7fc700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff108098780 0x7ff10809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:21.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.529+0000 7ff12a58e700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff108098780 0x7ff10809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:21.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.529+0000 7ff11a7fc700 1 -- 172.21.15.94:0/3655454307 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7ff11c0bfce0 con 0x7ff1240a6fc0 2023-09-30T00:20:21.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.530+0000 7ff12a58e700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff108098780 0x7ff10809ac30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff120007fd0 tx=0x7ff120001650 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:21.533 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.532+0000 7ff11a7fc700 1 -- 172.21.15.94:0/3655454307 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff11c08d320 con 0x7ff1240a6fc0 2023-09-30T00:20:21.757 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.756+0000 7ff12c7f2700 1 -- 172.21.15.94:0/3655454307 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff12406ecb0 con 0x7ff1240a6fc0 2023-09-30T00:20:21.757 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.757+0000 7ff11a7fc700 1 -- 172.21.15.94:0/3655454307 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff11c017020 con 0x7ff1240a6fc0 2023-09-30T00:20:21.757 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:21.758 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:21.759 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.758+0000 7ff12c7f2700 10 mgrc shutdown 2023-09-30T00:20:21.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.759+0000 7ff12c7f2700 1 -- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff108098780 msgr2=0x7ff10809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:21.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.759+0000 7ff12c7f2700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff108098780 0x7ff10809ac30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff120007fd0 tx=0x7ff120001650 comp rx=0 tx=0).stop 2023-09-30T00:20:21.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.759+0000 7ff12c7f2700 1 -- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1240a6fc0 msgr2=0x7ff12414e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:21.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.759+0000 7ff12c7f2700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1240a6fc0 0x7ff12414e1d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff11c001eb0 tx=0x7ff11c007cb0 comp rx=0 tx=0).stop 2023-09-30T00:20:21.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.759+0000 7ff12c7f2700 1 -- 172.21.15.94:0/3655454307 shutdown_connections 2023-09-30T00:20:21.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.759+0000 7ff12c7f2700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff108098780 0x7ff10809ac30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:21.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.759+0000 7ff12c7f2700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1240a6fc0 0x7ff12414e1d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:21.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.759+0000 7ff12c7f2700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1240a4470 0x7ff1240aaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:21.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.759+0000 7ff12c7f2700 1 --2- 172.21.15.94:0/3655454307 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff1240a4840 0x7ff1240ac4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:21.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.759+0000 7ff12c7f2700 1 -- 172.21.15.94:0/3655454307 >> 172.21.15.94:0/3655454307 conn(0x7ff12406cbf0 msgr2=0x7ff124143f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:21.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.759+0000 7ff12c7f2700 1 -- 172.21.15.94:0/3655454307 shutdown_connections 2023-09-30T00:20:21.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:21.759+0000 7ff12c7f2700 1 -- 172.21.15.94:0/3655454307 wait complete. 2023-09-30T00:20:24.771 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:24.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.896+0000 7f635ffff700 1 Processor -- start 2023-09-30T00:20:24.897 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.896+0000 7f635ffff700 1 -- start start 2023-09-30T00:20:24.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635ffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6360070540 0x7f636009d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:24.902 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635ffff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f636009f850 0x7f6360140d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:24.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635ffff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f636009c420 0x7f63601412b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:24.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635ffff700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63601409b0 con 0x7f636009f850 2023-09-30T00:20:24.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635ffff700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6360070910 con 0x7f636009c420 2023-09-30T00:20:24.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635ffff700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636009e270 con 0x7f6360070540 2023-09-30T00:20:24.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635e7fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f636009f850 0x7f6360140d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:24.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635effd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6360070540 0x7f636009d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:24.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635effd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6360070540 0x7f636009d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:60510/0 (socket says 172.21.15.94:60510) 2023-09-30T00:20:24.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635e7fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f636009f850 0x7f6360140d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43038/0 (socket says 172.21.15.94:43038) 2023-09-30T00:20:24.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635effd700 1 -- 172.21.15.94:0/950202978 learned_addr learned my addr 172.21.15.94:0/950202978 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:24.903 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635effd700 1 -- 172.21.15.94:0/950202978 >> v2:172.21.15.121:3300/0 conn(0x7f636009c420 msgr2=0x7f63601412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:24.904 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635f7fe700 1 --2- 172.21.15.94:0/950202978 >> v2:172.21.15.121:3300/0 conn(0x7f636009c420 0x7f63601412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:24.904 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635effd700 1 --2- 172.21.15.94:0/950202978 >> v2:172.21.15.121:3300/0 conn(0x7f636009c420 0x7f63601412b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:24.904 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635effd700 1 -- 172.21.15.94:0/950202978 >> v2:172.21.15.94:3300/0 conn(0x7f636009f850 msgr2=0x7f6360140d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:24.904 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635effd700 1 --2- 172.21.15.94:0/950202978 >> v2:172.21.15.94:3300/0 conn(0x7f636009f850 0x7f6360140d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:24.904 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635effd700 1 -- 172.21.15.94:0/950202978 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636009db90 con 0x7f6360070540 2023-09-30T00:20:24.904 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635f7fe700 1 --2- 172.21.15.94:0/950202978 >> v2:172.21.15.121:3300/0 conn(0x7f636009c420 0x7f63601412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:20:24.904 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635e7fc700 1 --2- 172.21.15.94:0/950202978 >> v2:172.21.15.94:3300/0 conn(0x7f636009f850 0x7f6360140d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:24.904 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635effd700 1 --2- 172.21.15.94:0/950202978 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6360070540 0x7f636009d210 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f636006b110 tx=0x7f6358009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93ae4f2a62e804d0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:24.904 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635dffb700 1 -- 172.21.15.94:0/950202978 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635800e070 con 0x7f6360070540 2023-09-30T00:20:24.904 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635dffb700 1 -- 172.21.15.94:0/950202978 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635800ba20 con 0x7f6360070540 2023-09-30T00:20:24.904 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.897+0000 7f635dffb700 1 -- 172.21.15.94:0/950202978 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63580092e0 con 0x7f6360070540 2023-09-30T00:20:24.904 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.898+0000 7f635ffff700 1 -- 172.21.15.94:0/950202978 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6360070540 msgr2=0x7f636009d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:24.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.898+0000 7f635ffff700 1 --2- 172.21.15.94:0/950202978 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6360070540 0x7f636009d210 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f636006b110 tx=0x7f6358009840 comp rx=0 tx=0).stop 2023-09-30T00:20:24.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.898+0000 7f635ffff700 1 -- 172.21.15.94:0/950202978 shutdown_connections 2023-09-30T00:20:24.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.898+0000 7f635ffff700 1 --2- 172.21.15.94:0/950202978 >> v2:172.21.15.121:3300/0 conn(0x7f636009c420 0x7f63601412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:24.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.898+0000 7f635ffff700 1 --2- 172.21.15.94:0/950202978 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6360070540 0x7f636009d210 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:24.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.898+0000 7f635ffff700 1 --2- 172.21.15.94:0/950202978 >> v2:172.21.15.94:3300/0 conn(0x7f636009f850 0x7f6360140d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:24.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.898+0000 7f635ffff700 1 -- 172.21.15.94:0/950202978 >> 172.21.15.94:0/950202978 conn(0x7f63600a35a0 msgr2=0x7f63600a39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:24.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.898+0000 7f635ffff700 1 -- 172.21.15.94:0/950202978 shutdown_connections 2023-09-30T00:20:24.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.898+0000 7f635ffff700 1 -- 172.21.15.94:0/950202978 wait complete. 2023-09-30T00:20:24.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.899+0000 7f635ffff700 1 Processor -- start 2023-09-30T00:20:24.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.899+0000 7f635ffff700 1 -- start start 2023-09-30T00:20:24.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635ffff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6360070540 0x7f6360142ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:24.906 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f636009c420 0x7f636013bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:24.906 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635ffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f636009f850 0x7f636013bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:24.906 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635ffff700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6360142750 con 0x7f6360070540 2023-09-30T00:20:24.906 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635ffff700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63601424e0 con 0x7f636009c420 2023-09-30T00:20:24.906 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635ffff700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6360140200 con 0x7f636009f850 2023-09-30T00:20:24.906 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6360070540 0x7f6360142ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:24.906 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6360070540 0x7f6360142ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43048/0 (socket says 172.21.15.94:43048) 2023-09-30T00:20:24.906 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635effd700 1 -- 172.21.15.94:0/238660346 learned_addr learned my addr 172.21.15.94:0/238660346 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:24.906 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635f7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f636009f850 0x7f636013bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:24.906 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635e7fc700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f636009c420 0x7f636013bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:24.906 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635effd700 1 -- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f636009f850 msgr2=0x7f636013bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:24.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635effd700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f636009f850 0x7f636013bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:24.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635effd700 1 -- 172.21.15.94:0/238660346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f636009c420 msgr2=0x7f636013bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:24.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635effd700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f636009c420 0x7f636013bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:24.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635effd700 1 -- 172.21.15.94:0/238660346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6354009420 con 0x7f6360070540 2023-09-30T00:20:24.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635f7fe700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f636009f850 0x7f636013bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:24.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f635effd700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6360070540 0x7f6360142ec0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f636009e3b0 tx=0x7f6358018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:24.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.900+0000 7f6346ffd700 1 -- 172.21.15.94:0/238660346 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6358008b80 con 0x7f6360070540 2023-09-30T00:20:24.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.901+0000 7f6346ffd700 1 -- 172.21.15.94:0/238660346 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6358008d00 con 0x7f6360070540 2023-09-30T00:20:24.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.901+0000 7f6346ffd700 1 -- 172.21.15.94:0/238660346 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63580266b0 con 0x7f6360070540 2023-09-30T00:20:24.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.901+0000 7f635ffff700 1 -- 172.21.15.94:0/238660346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6358009ae0 con 0x7f6360070540 2023-09-30T00:20:24.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.901+0000 7f635ffff700 1 -- 172.21.15.94:0/238660346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6360147780 con 0x7f6360070540 2023-09-30T00:20:24.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.901+0000 7f635ffff700 1 -- 172.21.15.94:0/238660346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636004c380 con 0x7f6360070540 2023-09-30T00:20:24.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.902+0000 7f6346ffd700 1 -- 172.21.15.94:0/238660346 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6358003040 con 0x7f6360070540 2023-09-30T00:20:24.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.902+0000 7f6346ffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:24.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.903+0000 7f6346ffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:24.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.903+0000 7f6346ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:24.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.903+0000 7f6346ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:24.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.903+0000 7f6346ffd700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f634c098780 0x7f634c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:24.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.903+0000 7f635e7fc700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f634c098780 0x7f634c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:24.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.903+0000 7f6346ffd700 1 -- 172.21.15.94:0/238660346 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f63580c22d0 con 0x7f6360070540 2023-09-30T00:20:24.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.903+0000 7f635e7fc700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f634c098780 0x7f634c09ac30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6354006f00 tx=0x7f6354008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:24.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:24.907+0000 7f6346ffd700 1 -- 172.21.15.94:0/238660346 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f635808f9a0 con 0x7f6360070540 2023-09-30T00:20:25.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.128+0000 7f635ffff700 1 -- 172.21.15.94:0/238660346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f63601417f0 con 0x7f6360070540 2023-09-30T00:20:25.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.128+0000 7f6346ffd700 1 -- 172.21.15.94:0/238660346 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6358092fc0 con 0x7f6360070540 2023-09-30T00:20:25.129 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:25.130 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:25.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 10 mgrc shutdown 2023-09-30T00:20:25.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 1 -- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f634c098780 msgr2=0x7f634c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:25.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f634c098780 0x7f634c09ac30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6354006f00 tx=0x7f6354008040 comp rx=0 tx=0).stop 2023-09-30T00:20:25.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 1 -- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6360070540 msgr2=0x7f6360142ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:25.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6360070540 0x7f6360142ec0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f636009e3b0 tx=0x7f6358018000 comp rx=0 tx=0).stop 2023-09-30T00:20:25.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 1 -- 172.21.15.94:0/238660346 shutdown_connections 2023-09-30T00:20:25.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f634c098780 0x7f634c09ac30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:25.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f636009f850 0x7f636013bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:25.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6360070540 0x7f6360142ec0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:25.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 1 --2- 172.21.15.94:0/238660346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f636009c420 0x7f636013bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:25.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 1 -- 172.21.15.94:0/238660346 >> 172.21.15.94:0/238660346 conn(0x7f63600a35a0 msgr2=0x7f636006e560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:25.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 1 -- 172.21.15.94:0/238660346 shutdown_connections 2023-09-30T00:20:25.132 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:25.130+0000 7f635ffff700 1 -- 172.21.15.94:0/238660346 wait complete. 2023-09-30T00:20:28.144 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:28.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.265+0000 7f7c7e376700 1 Processor -- start 2023-09-30T00:20:28.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.265+0000 7f7c7e376700 1 -- start start 2023-09-30T00:20:28.268 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c7e376700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 0x7f7c780a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:28.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c7e376700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7c780b7160 0x7f7c780a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:28.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c7e376700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c780a1310 con 0x7f7c7813fd70 2023-09-30T00:20:28.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c7e376700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c78071020 con 0x7f7c780b7160 2023-09-30T00:20:28.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c7e376700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c780b76a0 con 0x7f7c78070c50 2023-09-30T00:20:28.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c77fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 0x7f7c780a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:28.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c777fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7c780b7160 0x7f7c780a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:28.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c7c913700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7c7813fd70 0x7f7c780b4dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43888/0 (socket says 172.21.15.94:43888) 2023-09-30T00:20:28.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c7c913700 1 -- 172.21.15.94:0/910071087 learned_addr learned my addr 172.21.15.94:0/910071087 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:20:28.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c777fe700 1 -- 172.21.15.94:0/910071087 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 msgr2=0x7f7c780a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:28.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c777fe700 1 --2- 172.21.15.94:0/910071087 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 0x7f7c780a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:28.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c777fe700 1 -- 172.21.15.94:0/910071087 >> v1:172.21.15.94:6789/0 conn(0x7f7c7813fd70 legacy=0x7f7c780b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:28.269 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c777fe700 1 -- 172.21.15.94:0/910071087 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c780b7940 con 0x7f7c780b7160 2023-09-30T00:20:28.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c76ffd700 1 -- 172.21.15.94:0/910071087 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696662446 0 0) 0x7f7c70001490 con 0x7f7c7813fd70 2023-09-30T00:20:28.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c77fff700 1 --2- 172.21.15.94:0/910071087 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 0x7f7c780a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:28.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c777fe700 1 --2- 172.21.15.94:0/910071087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c780b7160 0x7f7c780a2530 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0098e0 tx=0x7f7c6c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af23b473e5ac465f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:28.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c76ffd700 1 -- 172.21.15.94:0/910071087 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c00b6d0 con 0x7f7c780b7160 2023-09-30T00:20:28.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c76ffd700 1 -- 172.21.15.94:0/910071087 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c00b830 con 0x7f7c780b7160 2023-09-30T00:20:28.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.266+0000 7f7c76ffd700 1 -- 172.21.15.94:0/910071087 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c00bbd0 con 0x7f7c780b7160 2023-09-30T00:20:28.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.267+0000 7f7c7e376700 1 -- 172.21.15.94:0/910071087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c780b7160 msgr2=0x7f7c780a2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:28.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.267+0000 7f7c7e376700 1 --2- 172.21.15.94:0/910071087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c780b7160 0x7f7c780a2530 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0098e0 tx=0x7f7c6c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:20:28.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.267+0000 7f7c7e376700 1 -- 172.21.15.94:0/910071087 shutdown_connections 2023-09-30T00:20:28.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.267+0000 7f7c7e376700 1 --2- 172.21.15.94:0/910071087 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 0x7f7c780a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:28.270 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.267+0000 7f7c7e376700 1 --2- 172.21.15.94:0/910071087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c780b7160 0x7f7c780a2530 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:28.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.267+0000 7f7c7e376700 1 -- 172.21.15.94:0/910071087 >> 172.21.15.94:0/910071087 conn(0x7f7c7806a090 msgr2=0x7f7c780702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:28.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.267+0000 7f7c7e376700 1 -- 172.21.15.94:0/910071087 shutdown_connections 2023-09-30T00:20:28.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.267+0000 7f7c7e376700 1 -- 172.21.15.94:0/910071087 wait complete. 2023-09-30T00:20:28.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.268+0000 7f7c7e376700 1 Processor -- start 2023-09-30T00:20:28.271 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.268+0000 7f7c7e376700 1 -- start start 2023-09-30T00:20:28.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c7e376700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 0x7f7c780ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:28.273 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c7e376700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c780b7160 0x7f7c78150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:28.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c7e376700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c7813fd70 0x7f7c781510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:28.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c7e376700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c78151620 con 0x7f7c7813fd70 2023-09-30T00:20:28.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c7e376700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c78150780 con 0x7f7c780b7160 2023-09-30T00:20:28.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c7e376700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c780b3be0 con 0x7f7c78070c50 2023-09-30T00:20:28.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c77fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 0x7f7c780ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:28.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c77fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 0x7f7c780ac1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:60554/0 (socket says 172.21.15.94:60554) 2023-09-30T00:20:28.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c7c913700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c7813fd70 0x7f7c781510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:28.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c77fff700 1 -- 172.21.15.94:0/2865912728 learned_addr learned my addr 172.21.15.94:0/2865912728 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:28.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c77fff700 1 -- 172.21.15.94:0/2865912728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c780b7160 msgr2=0x7f7c78150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:28.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c77fff700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c780b7160 0x7f7c78150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:28.274 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c77fff700 1 -- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c7813fd70 msgr2=0x7f7c781510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:28.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c77fff700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c7813fd70 0x7f7c781510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:28.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c77fff700 1 -- 172.21.15.94:0/2865912728 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c68009420 con 0x7f7c78070c50 2023-09-30T00:20:28.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c7c913700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c7813fd70 0x7f7c781510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:28.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c77fff700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 0x7f7c780ac1f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800ecb0 tx=0x7f7c680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:28.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c5ffff700 1 -- 172.21.15.94:0/2865912728 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6800c950 con 0x7f7c78070c50 2023-09-30T00:20:28.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c5ffff700 1 -- 172.21.15.94:0/2865912728 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6800cab0 con 0x7f7c78070c50 2023-09-30T00:20:28.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c7e376700 1 -- 172.21.15.94:0/2865912728 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6c009420 con 0x7f7c78070c50 2023-09-30T00:20:28.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c5ffff700 1 -- 172.21.15.94:0/2865912728 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6800ce30 con 0x7f7c78070c50 2023-09-30T00:20:28.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.269+0000 7f7c7e376700 1 -- 172.21.15.94:0/2865912728 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c780a8190 con 0x7f7c78070c50 2023-09-30T00:20:28.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.270+0000 7f7c7e376700 1 -- 172.21.15.94:0/2865912728 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c780ad030 con 0x7f7c78070c50 2023-09-30T00:20:28.275 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.271+0000 7f7c5ffff700 1 -- 172.21.15.94:0/2865912728 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f7c68013680 con 0x7f7c78070c50 2023-09-30T00:20:28.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.271+0000 7f7c5ffff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:28.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.272+0000 7f7c5ffff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:28.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.272+0000 7f7c5ffff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:28.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.272+0000 7f7c5ffff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:28.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.272+0000 7f7c5ffff700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7c64098780 0x7f7c6409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:28.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.272+0000 7f7c777fe700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7c64098780 0x7f7c6409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:28.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.272+0000 7f7c5ffff700 1 -- 172.21.15.94:0/2865912728 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f7c680c0e10 con 0x7f7c78070c50 2023-09-30T00:20:28.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.272+0000 7f7c777fe700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7c64098780 0x7f7c6409ac30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7c7813ca00 tx=0x7f7c6c017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:28.276 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.275+0000 7f7c5ffff700 1 -- 172.21.15.94:0/2865912728 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c6808e4e0 con 0x7f7c78070c50 2023-09-30T00:20:28.497 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.495+0000 7f7c7e376700 1 -- 172.21.15.94:0/2865912728 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7c780ad030 con 0x7f7c78070c50 2023-09-30T00:20:28.497 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.496+0000 7f7c5ffff700 1 -- 172.21.15.94:0/2865912728 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7c780ad030 con 0x7f7c78070c50 2023-09-30T00:20:28.497 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:28.497 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:28.499 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 10 mgrc shutdown 2023-09-30T00:20:28.500 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 1 -- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7c64098780 msgr2=0x7f7c6409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:28.500 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7c64098780 0x7f7c6409ac30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7c7813ca00 tx=0x7f7c6c017000 comp rx=0 tx=0).stop 2023-09-30T00:20:28.500 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 1 -- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 msgr2=0x7f7c780ac1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:28.500 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 0x7f7c780ac1f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800ecb0 tx=0x7f7c680091a0 comp rx=0 tx=0).stop 2023-09-30T00:20:28.500 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 1 -- 172.21.15.94:0/2865912728 shutdown_connections 2023-09-30T00:20:28.500 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7c64098780 0x7f7c6409ac30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:28.500 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c78070c50 0x7f7c780ac1f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:28.500 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c7813fd70 0x7f7c781510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:28.500 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 1 --2- 172.21.15.94:0/2865912728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c780b7160 0x7f7c78150ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:28.500 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 1 -- 172.21.15.94:0/2865912728 >> 172.21.15.94:0/2865912728 conn(0x7f7c7806a090 msgr2=0x7f7c780b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:28.501 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 1 -- 172.21.15.94:0/2865912728 shutdown_connections 2023-09-30T00:20:28.501 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:28.498+0000 7f7c7e376700 1 -- 172.21.15.94:0/2865912728 wait complete. 2023-09-30T00:20:31.512 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:31.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.634+0000 7f80b5697700 1 Processor -- start 2023-09-30T00:20:31.636 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.634+0000 7f80b5697700 1 -- start start 2023-09-30T00:20:31.638 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80b5697700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f80b00a51c0 0x7f80b0143e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:31.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80b5697700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80b00a5620 0x7f80b0144390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:31.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80b5697700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b0067e90 con 0x7f80b0067ac0 2023-09-30T00:20:31.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80b5697700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b01448d0 con 0x7f80b00a51c0 2023-09-30T00:20:31.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80b5697700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b006a230 con 0x7f80b00a5620 2023-09-30T00:20:31.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80aeffd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f80b0067ac0 0x7f80b0136950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43896/0 (socket says 172.21.15.94:43896) 2023-09-30T00:20:31.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80af7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80b00a5620 0x7f80b0144390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:31.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80aeffd700 1 -- 172.21.15.94:0/2663464609 learned_addr learned my addr 172.21.15.94:0/2663464609 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:20:31.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80a67fc700 1 --2- 172.21.15.94:0/2663464609 >> v2:172.21.15.121:3300/0 conn(0x7f80b00a51c0 0x7f80b0143e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:31.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80ae7fc700 1 -- 172.21.15.94:0/2663464609 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986928824 0 0) 0x7f80b0067e90 con 0x7f80b0067ac0 2023-09-30T00:20:31.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80ae7fc700 1 -- 172.21.15.94:0/2663464609 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f809c003570 con 0x7f80b0067ac0 2023-09-30T00:20:31.639 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80af7fe700 1 -- 172.21.15.94:0/2663464609 >> v2:172.21.15.121:3300/0 conn(0x7f80b00a51c0 msgr2=0x7f80b0143e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:31.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80af7fe700 1 --2- 172.21.15.94:0/2663464609 >> v2:172.21.15.121:3300/0 conn(0x7f80b00a51c0 0x7f80b0143e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:31.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80af7fe700 1 -- 172.21.15.94:0/2663464609 >> v1:172.21.15.94:6789/0 conn(0x7f80b0067ac0 legacy=0x7f80b0136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:31.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.635+0000 7f80af7fe700 1 -- 172.21.15.94:0/2663464609 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b0144ae0 con 0x7f80b00a5620 2023-09-30T00:20:31.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.636+0000 7f80af7fe700 1 --2- 172.21.15.94:0/2663464609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80b00a5620 0x7f80b0144390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009e90 tx=0x7f80a8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd08674c114d22c5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:31.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.636+0000 7f80ae7fc700 1 -- 172.21.15.94:0/2663464609 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a800e070 con 0x7f80b00a5620 2023-09-30T00:20:31.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.636+0000 7f80ae7fc700 1 -- 172.21.15.94:0/2663464609 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a800bbf0 con 0x7f80b00a5620 2023-09-30T00:20:31.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.636+0000 7f80ae7fc700 1 -- 172.21.15.94:0/2663464609 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8012430 con 0x7f80b00a5620 2023-09-30T00:20:31.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.636+0000 7f80b5697700 1 -- 172.21.15.94:0/2663464609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80b00a5620 msgr2=0x7f80b0144390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:31.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.636+0000 7f80b5697700 1 --2- 172.21.15.94:0/2663464609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80b00a5620 0x7f80b0144390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009e90 tx=0x7f80a8009780 comp rx=0 tx=0).stop 2023-09-30T00:20:31.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.636+0000 7f80b5697700 1 -- 172.21.15.94:0/2663464609 shutdown_connections 2023-09-30T00:20:31.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.636+0000 7f80b5697700 1 --2- 172.21.15.94:0/2663464609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80b00a5620 0x7f80b0144390 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:31.640 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.636+0000 7f80b5697700 1 --2- 172.21.15.94:0/2663464609 >> v2:172.21.15.121:3300/0 conn(0x7f80b00a51c0 0x7f80b0143e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:31.641 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.636+0000 7f80b5697700 1 -- 172.21.15.94:0/2663464609 >> 172.21.15.94:0/2663464609 conn(0x7f80b006cb10 msgr2=0x7f80b006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:31.641 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.636+0000 7f80b5697700 1 -- 172.21.15.94:0/2663464609 shutdown_connections 2023-09-30T00:20:31.641 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.636+0000 7f80b5697700 1 -- 172.21.15.94:0/2663464609 wait complete. 2023-09-30T00:20:31.641 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.637+0000 7f80b5697700 1 Processor -- start 2023-09-30T00:20:31.641 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.637+0000 7f80b5697700 1 -- start start 2023-09-30T00:20:31.643 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80b5697700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b0067ac0 0x7f80b00a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:31.643 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80b5697700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80b00a51c0 0x7f80b009f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:31.643 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80b5697700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80b00a5620 0x7f80b0153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:31.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80b5697700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b00a0e50 con 0x7f80b00a51c0 2023-09-30T00:20:31.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80b5697700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b00a1ab0 con 0x7f80b0067ac0 2023-09-30T00:20:31.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80b5697700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b0135fb0 con 0x7f80b00a5620 2023-09-30T00:20:31.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80a67fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80b00a51c0 0x7f80b009f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:31.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80a67fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80b00a51c0 0x7f80b009f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43074/0 (socket says 172.21.15.94:43074) 2023-09-30T00:20:31.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80a67fc700 1 -- 172.21.15.94:0/1036074912 learned_addr learned my addr 172.21.15.94:0/1036074912 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:31.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80af7fe700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80b00a5620 0x7f80b0153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:31.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80aeffd700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b0067ac0 0x7f80b00a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:31.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80a67fc700 1 -- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80b00a5620 msgr2=0x7f80b0153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:31.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80a67fc700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80b00a5620 0x7f80b0153440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:31.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80a67fc700 1 -- 172.21.15.94:0/1036074912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b0067ac0 msgr2=0x7f80b00a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:31.644 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80a67fc700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b0067ac0 0x7f80b00a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80a67fc700 1 -- 172.21.15.94:0/1036074912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a8009a20 con 0x7f80b00a51c0 2023-09-30T00:20:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80af7fe700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80b00a5620 0x7f80b0153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80a67fc700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80b00a51c0 0x7f80b009f050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f809800d890 tx=0x7f8098008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80a77fe700 1 -- 172.21.15.94:0/1036074912 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80980125f0 con 0x7f80b00a51c0 2023-09-30T00:20:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80a77fe700 1 -- 172.21.15.94:0/1036074912 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8098012750 con 0x7f80b00a51c0 2023-09-30T00:20:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.638+0000 7f80b5697700 1 -- 172.21.15.94:0/1036074912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b00a1d50 con 0x7f80b00a51c0 2023-09-30T00:20:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.639+0000 7f80a77fe700 1 -- 172.21.15.94:0/1036074912 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098012af0 con 0x7f80b00a51c0 2023-09-30T00:20:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.639+0000 7f80b5697700 1 -- 172.21.15.94:0/1036074912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b009f590 con 0x7f80b00a51c0 2023-09-30T00:20:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.639+0000 7f80b5697700 1 -- 172.21.15.94:0/1036074912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b01361d0 con 0x7f80b00a51c0 2023-09-30T00:20:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.640+0000 7f80a77fe700 1 -- 172.21.15.94:0/1036074912 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f809801e460 con 0x7f80b00a51c0 2023-09-30T00:20:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.640+0000 7f80a77fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:31.645 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.641+0000 7f80a77fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:31.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.641+0000 7f80a77fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:31.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.641+0000 7f80a77fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:31.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.641+0000 7f80a77fe700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8090098780 0x7f809009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:31.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.641+0000 7f80aeffd700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8090098780 0x7f809009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:31.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.641+0000 7f80a77fe700 1 -- 172.21.15.94:0/1036074912 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f80980c0950 con 0x7f80b00a51c0 2023-09-30T00:20:31.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.641+0000 7f80aeffd700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8090098780 0x7f809009ac30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f80a0007fd0 tx=0x7f80a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:31.646 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.645+0000 7f80a77fe700 1 -- 172.21.15.94:0/1036074912 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f80b01361d0 con 0x7f80b00a51c0 2023-09-30T00:20:31.870 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.868+0000 7f80b5697700 1 -- 172.21.15.94:0/1036074912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f80b01361d0 con 0x7f80b00a51c0 2023-09-30T00:20:31.870 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.869+0000 7f80a77fe700 1 -- 172.21.15.94:0/1036074912 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f80b01361d0 con 0x7f80b00a51c0 2023-09-30T00:20:31.870 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:31.871 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:31.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 10 mgrc shutdown 2023-09-30T00:20:31.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 1 -- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8090098780 msgr2=0x7f809009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:31.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8090098780 0x7f809009ac30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f80a0007fd0 tx=0x7f80a0009040 comp rx=0 tx=0).stop 2023-09-30T00:20:31.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 1 -- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80b00a51c0 msgr2=0x7f80b009f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:31.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80b00a51c0 0x7f80b009f050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f809800d890 tx=0x7f8098008040 comp rx=0 tx=0).stop 2023-09-30T00:20:31.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 1 -- 172.21.15.94:0/1036074912 shutdown_connections 2023-09-30T00:20:31.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f8090098780 0x7f809009ac30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:31.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80b00a5620 0x7f80b0153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:31.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b0067ac0 0x7f80b00a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:31.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 1 --2- 172.21.15.94:0/1036074912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80b00a51c0 0x7f80b009f050 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:31.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 1 -- 172.21.15.94:0/1036074912 >> 172.21.15.94:0/1036074912 conn(0x7f80b006cb10 msgr2=0x7f80b0136720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:31.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 1 -- 172.21.15.94:0/1036074912 shutdown_connections 2023-09-30T00:20:31.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:31.871+0000 7f80b5697700 1 -- 172.21.15.94:0/1036074912 wait complete. 2023-09-30T00:20:34.884 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.008+0000 7fdc14e32700 1 Processor -- start 2023-09-30T00:20:35.009 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.008+0000 7fdc14e32700 1 -- start start 2023-09-30T00:20:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc14e32700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdc1009de30 0x7fdc10140b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc14e32700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdc1006eab0 0x7fdc1009f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc14e32700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1009fdc0 con 0x7fdc100a2ef0 2023-09-30T00:20:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc14e32700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc10141180 con 0x7fdc1006eab0 2023-09-30T00:20:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc14e32700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc10141440 con 0x7fdc1009de30 2023-09-30T00:20:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0dd9b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdc1006eab0 0x7fdc1009f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0dd9b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdc1006eab0 0x7fdc1009f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56114/0 (socket says 172.21.15.94:56114) 2023-09-30T00:20:35.012 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0dd9b700 1 -- 172.21.15.94:0/2012603162 learned_addr learned my addr 172.21.15.94:0/2012603162 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0e59c700 1 --2- 172.21.15.94:0/2012603162 >> v2:172.21.15.121:3300/0 conn(0x7fdc1009de30 0x7fdc10140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0dd9b700 1 -- 172.21.15.94:0/2012603162 >> v2:172.21.15.121:3300/0 conn(0x7fdc1009de30 msgr2=0x7fdc10140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0dd9b700 1 --2- 172.21.15.94:0/2012603162 >> v2:172.21.15.121:3300/0 conn(0x7fdc1009de30 0x7fdc10140b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0dd9b700 1 -- 172.21.15.94:0/2012603162 >> v1:172.21.15.121:6789/0 conn(0x7fdc100a2ef0 legacy=0x7fdc100a4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0dd9b700 1 -- 172.21.15.94:0/2012603162 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc10141700 con 0x7fdc1006eab0 2023-09-30T00:20:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0e59c700 1 --2- 172.21.15.94:0/2012603162 >> v2:172.21.15.121:3300/0 conn(0x7fdc1009de30 0x7fdc10140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:20:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0dd9b700 1 --2- 172.21.15.94:0/2012603162 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc1006eab0 0x7fdc1009f880 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdc100695b0 tx=0x7fdc04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af4cc268f3b13ee server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0d59a700 1 -- 172.21.15.94:0/2012603162 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0400b8b0 con 0x7fdc1006eab0 2023-09-30T00:20:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0d59a700 1 -- 172.21.15.94:0/2012603162 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0400ba10 con 0x7fdc1006eab0 2023-09-30T00:20:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.009+0000 7fdc0d59a700 1 -- 172.21.15.94:0/2012603162 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0400bdb0 con 0x7fdc1006eab0 2023-09-30T00:20:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.010+0000 7fdc14e32700 1 -- 172.21.15.94:0/2012603162 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc1006eab0 msgr2=0x7fdc1009f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:35.013 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.010+0000 7fdc14e32700 1 --2- 172.21.15.94:0/2012603162 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc1006eab0 0x7fdc1009f880 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdc100695b0 tx=0x7fdc04009250 comp rx=0 tx=0).stop 2023-09-30T00:20:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.010+0000 7fdc14e32700 1 -- 172.21.15.94:0/2012603162 shutdown_connections 2023-09-30T00:20:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.010+0000 7fdc14e32700 1 --2- 172.21.15.94:0/2012603162 >> v2:172.21.15.121:3300/0 conn(0x7fdc1009de30 0x7fdc10140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.010+0000 7fdc14e32700 1 --2- 172.21.15.94:0/2012603162 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc1006eab0 0x7fdc1009f880 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.010+0000 7fdc14e32700 1 -- 172.21.15.94:0/2012603162 >> 172.21.15.94:0/2012603162 conn(0x7fdc100b3520 msgr2=0x7fdc10067620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.010+0000 7fdc14e32700 1 -- 172.21.15.94:0/2012603162 shutdown_connections 2023-09-30T00:20:35.014 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.010+0000 7fdc14e32700 1 -- 172.21.15.94:0/2012603162 wait complete. 2023-09-30T00:20:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.011+0000 7fdc14e32700 1 Processor -- start 2023-09-30T00:20:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.011+0000 7fdc14e32700 1 -- start start 2023-09-30T00:20:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.011+0000 7fdc14e32700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc1006eab0 0x7fdc1013f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc14e32700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc1009de30 0x7fdc10096320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc14e32700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdc100a2ef0 0x7fdc1006d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc14e32700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc10140860 con 0x7fdc1006eab0 2023-09-30T00:20:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc14e32700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1014d4c0 con 0x7fdc1009de30 2023-09-30T00:20:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc14e32700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc10140230 con 0x7fdc100a2ef0 2023-09-30T00:20:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0e59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc1006eab0 0x7fdc1013f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:35.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0e59c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc1006eab0 0x7fdc1013f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56122/0 (socket says 172.21.15.94:56122) 2023-09-30T00:20:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0e59c700 1 -- 172.21.15.94:0/2973224192 learned_addr learned my addr 172.21.15.94:0/2973224192 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0ed9d700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdc100a2ef0 0x7fdc1006d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0dd9b700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc1009de30 0x7fdc10096320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0e59c700 1 -- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdc100a2ef0 msgr2=0x7fdc1006d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0e59c700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdc100a2ef0 0x7fdc1006d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0e59c700 1 -- 172.21.15.94:0/2973224192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc1009de30 msgr2=0x7fdc10096320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0e59c700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc1009de30 0x7fdc10096320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0e59c700 1 -- 172.21.15.94:0/2973224192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc040094f0 con 0x7fdc1006eab0 2023-09-30T00:20:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0ed9d700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdc100a2ef0 0x7fdc1006d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0dd9b700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc1009de30 0x7fdc10096320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:20:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc0e59c700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc1006eab0 0x7fdc1013f9d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdc0800a400 tx=0x7fdc08007910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdbfe7fc700 1 -- 172.21.15.94:0/2973224192 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0800ae20 con 0x7fdc1006eab0 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdbfe7fc700 1 -- 172.21.15.94:0/2973224192 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0800d650 con 0x7fdc1006eab0 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc14e32700 1 -- 172.21.15.94:0/2973224192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1014d760 con 0x7fdc1006eab0 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdbfe7fc700 1 -- 172.21.15.94:0/2973224192 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0800d9f0 con 0x7fdc1006eab0 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.012+0000 7fdc14e32700 1 -- 172.21.15.94:0/2973224192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc10095c50 con 0x7fdc1006eab0 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.013+0000 7fdc14e32700 1 -- 172.21.15.94:0/2973224192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc100a5720 con 0x7fdc1006eab0 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.014+0000 7fdbfe7fc700 1 -- 172.21.15.94:0/2973224192 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fdc08015440 con 0x7fdc1006eab0 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.014+0000 7fdbfe7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.015+0000 7fdbfe7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.015+0000 7fdbfe7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.015+0000 7fdbfe7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.015+0000 7fdbfe7fc700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdbf00987b0 0x7fdbf009ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:35.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.015+0000 7fdc0dd9b700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdbf00987b0 0x7fdbf009ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.015+0000 7fdbfe7fc700 1 -- 172.21.15.94:0/2973224192 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fdc080c13e0 con 0x7fdc1006eab0 2023-09-30T00:20:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.015+0000 7fdc0dd9b700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdbf00987b0 0x7fdbf009ac60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdc100695b0 tx=0x7fdc04016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:35.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.018+0000 7fdbfe7fc700 1 -- 172.21.15.94:0/2973224192 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdc0808eab0 con 0x7fdc1006eab0 2023-09-30T00:20:35.247 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.245+0000 7fdc14e32700 1 -- 172.21.15.94:0/2973224192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdc10071630 con 0x7fdc1006eab0 2023-09-30T00:20:35.247 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.246+0000 7fdbfe7fc700 1 -- 172.21.15.94:0/2973224192 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdc08024100 con 0x7fdc1006eab0 2023-09-30T00:20:35.247 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:35.248 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:35.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 10 mgrc shutdown 2023-09-30T00:20:35.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 1 -- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdbf00987b0 msgr2=0x7fdbf009ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:35.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdbf00987b0 0x7fdbf009ac60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdc100695b0 tx=0x7fdc04016000 comp rx=0 tx=0).stop 2023-09-30T00:20:35.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 1 -- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc1006eab0 msgr2=0x7fdc1013f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:35.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc1006eab0 0x7fdc1013f9d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdc0800a400 tx=0x7fdc08007910 comp rx=0 tx=0).stop 2023-09-30T00:20:35.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 1 -- 172.21.15.94:0/2973224192 shutdown_connections 2023-09-30T00:20:35.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdbf00987b0 0x7fdbf009ac60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:35.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdc100a2ef0 0x7fdc1006d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:35.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc1006eab0 0x7fdc1013f9d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:35.251 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 1 --2- 172.21.15.94:0/2973224192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc1009de30 0x7fdc10096320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:35.251 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 1 -- 172.21.15.94:0/2973224192 >> 172.21.15.94:0/2973224192 conn(0x7fdc100b3520 msgr2=0x7fdc1013dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:35.251 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 1 -- 172.21.15.94:0/2973224192 shutdown_connections 2023-09-30T00:20:35.251 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:35.248+0000 7fdc14e32700 1 -- 172.21.15.94:0/2973224192 wait complete. 2023-09-30T00:20:38.262 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:38.397 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.395+0000 7f24b6ef9700 1 Processor -- start 2023-09-30T00:20:38.397 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.395+0000 7f24b6ef9700 1 -- start start 2023-09-30T00:20:38.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24b6ef9700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24b00b7160 0x7f24b00a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:38.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24b6ef9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24b013fd70 0x7f24b01401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:38.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24b6ef9700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b00a1310 con 0x7f24b0070c50 2023-09-30T00:20:38.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24b6ef9700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b0071020 con 0x7f24b00b7160 2023-09-30T00:20:38.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24b6ef9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b00b76a0 con 0x7f24b013fd70 2023-09-30T00:20:38.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24a7fff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24b00b7160 0x7f24b00a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:38.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24b5496700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24b013fd70 0x7f24b01401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:38.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24b5496700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24b013fd70 0x7f24b01401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:47892/0 (socket says 172.21.15.94:47892) 2023-09-30T00:20:38.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24a7fff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24b00b7160 0x7f24b00a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56130/0 (socket says 172.21.15.94:56130) 2023-09-30T00:20:38.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24b5496700 1 -- 172.21.15.94:0/1850344057 learned_addr learned my addr 172.21.15.94:0/1850344057 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:38.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24a7fff700 1 -- 172.21.15.94:0/1850344057 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24b013fd70 msgr2=0x7f24b01401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:38.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24a7fff700 1 --2- 172.21.15.94:0/1850344057 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24b013fd70 0x7f24b01401d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:38.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24a7fff700 1 -- 172.21.15.94:0/1850344057 >> v1:172.21.15.121:6789/0 conn(0x7f24b0070c50 legacy=0x7f24b00a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:38.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24a7fff700 1 -- 172.21.15.94:0/1850344057 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b00b7940 con 0x7f24b00b7160 2023-09-30T00:20:38.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.396+0000 7f24b5496700 1 --2- 172.21.15.94:0/1850344057 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24b013fd70 0x7f24b01401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:38.400 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.397+0000 7f24a7fff700 1 --2- 172.21.15.94:0/1850344057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24b00b7160 0x7f24b00a0dd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f24a00098e0 tx=0x7f24a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ddf8bb4471c8271 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:38.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.397+0000 7f24a77fe700 1 -- 172.21.15.94:0/1850344057 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a000b690 con 0x7f24b00b7160 2023-09-30T00:20:38.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.397+0000 7f24a77fe700 1 -- 172.21.15.94:0/1850344057 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a000b7f0 con 0x7f24b00b7160 2023-09-30T00:20:38.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.397+0000 7f24a77fe700 1 -- 172.21.15.94:0/1850344057 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a000bb90 con 0x7f24b00b7160 2023-09-30T00:20:38.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.397+0000 7f24b6ef9700 1 -- 172.21.15.94:0/1850344057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24b00b7160 msgr2=0x7f24b00a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:38.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.397+0000 7f24b6ef9700 1 --2- 172.21.15.94:0/1850344057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24b00b7160 0x7f24b00a0dd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f24a00098e0 tx=0x7f24a00091a0 comp rx=0 tx=0).stop 2023-09-30T00:20:38.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.397+0000 7f24b6ef9700 1 -- 172.21.15.94:0/1850344057 shutdown_connections 2023-09-30T00:20:38.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.397+0000 7f24b6ef9700 1 --2- 172.21.15.94:0/1850344057 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24b013fd70 0x7f24b01401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:38.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.397+0000 7f24b6ef9700 1 --2- 172.21.15.94:0/1850344057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24b00b7160 0x7f24b00a0dd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:38.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.397+0000 7f24b6ef9700 1 -- 172.21.15.94:0/1850344057 >> 172.21.15.94:0/1850344057 conn(0x7f24b006a090 msgr2=0x7f24b00702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:38.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.397+0000 7f24b6ef9700 1 -- 172.21.15.94:0/1850344057 shutdown_connections 2023-09-30T00:20:38.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.397+0000 7f24b6ef9700 1 -- 172.21.15.94:0/1850344057 wait complete. 2023-09-30T00:20:38.404 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.398+0000 7f24b6ef9700 1 Processor -- start 2023-09-30T00:20:38.404 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.398+0000 7f24b6ef9700 1 -- start start 2023-09-30T00:20:38.404 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24b6ef9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24b0070c50 0x7f24b00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24b6ef9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24b00b7160 0x7f24b00a8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24b6ef9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24b013fd70 0x7f24b0143140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24b6ef9700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b0142bf0 con 0x7f24b00b7160 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24b6ef9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b0144c80 con 0x7f24b0070c50 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24b6ef9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b00a49d0 con 0x7f24b013fd70 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24a7fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24b00b7160 0x7f24b00a8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24b5496700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24b013fd70 0x7f24b0143140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24a7fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24b00b7160 0x7f24b00a8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56134/0 (socket says 172.21.15.94:56134) 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24a7fff700 1 -- 172.21.15.94:0/2350064286 learned_addr learned my addr 172.21.15.94:0/2350064286 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24b4c95700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24b0070c50 0x7f24b00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24a7fff700 1 -- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24b013fd70 msgr2=0x7f24b0143140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24a7fff700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24b013fd70 0x7f24b0143140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:38.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24a7fff700 1 -- 172.21.15.94:0/2350064286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24b0070c50 msgr2=0x7f24b00a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:38.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24a7fff700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24b0070c50 0x7f24b00a43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:38.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24a7fff700 1 -- 172.21.15.94:0/2350064286 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a8009ae0 con 0x7f24b00b7160 2023-09-30T00:20:38.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.399+0000 7f24b4c95700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24b0070c50 0x7f24b00a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:20:38.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.400+0000 7f24a7fff700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24b00b7160 0x7f24b00a8350 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f24a00093f0 tx=0x7f24a0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:38.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.400+0000 7f24a4ff9700 1 -- 172.21.15.94:0/2350064286 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0011c20 con 0x7f24b00b7160 2023-09-30T00:20:38.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.400+0000 7f24a4ff9700 1 -- 172.21.15.94:0/2350064286 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a0020070 con 0x7f24b00b7160 2023-09-30T00:20:38.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.400+0000 7f24b6ef9700 1 -- 172.21.15.94:0/2350064286 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a0009420 con 0x7f24b00b7160 2023-09-30T00:20:38.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.400+0000 7f24a4ff9700 1 -- 172.21.15.94:0/2350064286 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a001b450 con 0x7f24b00b7160 2023-09-30T00:20:38.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.400+0000 7f24b6ef9700 1 -- 172.21.15.94:0/2350064286 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b00a8bb0 con 0x7f24b00b7160 2023-09-30T00:20:38.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.401+0000 7f24b6ef9700 1 -- 172.21.15.94:0/2350064286 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b00a6000 con 0x7f24b00b7160 2023-09-30T00:20:38.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.401+0000 7f24a4ff9700 1 -- 172.21.15.94:0/2350064286 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f24a0022030 con 0x7f24b00b7160 2023-09-30T00:20:38.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.401+0000 7f24a4ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:38.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.402+0000 7f24a4ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:38.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.402+0000 7f24a4ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:38.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.402+0000 7f24a4ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:38.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.402+0000 7f24a4ff9700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f249c098780 0x7f249c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:38.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.402+0000 7f24b4c95700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f249c098780 0x7f249c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:38.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.402+0000 7f24a4ff9700 1 -- 172.21.15.94:0/2350064286 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f24a00c2020 con 0x7f24b00b7160 2023-09-30T00:20:38.407 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.402+0000 7f24b4c95700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f249c098780 0x7f249c09ac30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f24ac002880 tx=0x7f24ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:38.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.406+0000 7f24a4ff9700 1 -- 172.21.15.94:0/2350064286 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f24a008f6f0 con 0x7f24b00b7160 2023-09-30T00:20:38.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.627+0000 7f24b6ef9700 1 -- 172.21.15.94:0/2350064286 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f24b00a1310 con 0x7f24b00b7160 2023-09-30T00:20:38.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.628+0000 7f24a4ff9700 1 -- 172.21.15.94:0/2350064286 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f24b00a1310 con 0x7f24b00b7160 2023-09-30T00:20:38.629 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:38.630 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:38.631 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.629+0000 7f24b6ef9700 10 mgrc shutdown 2023-09-30T00:20:38.631 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.630+0000 7f24b6ef9700 1 -- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f249c098780 msgr2=0x7f249c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:38.631 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.630+0000 7f24b6ef9700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f249c098780 0x7f249c09ac30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f24ac002880 tx=0x7f24ac009040 comp rx=0 tx=0).stop 2023-09-30T00:20:38.632 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.630+0000 7f24b6ef9700 1 -- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24b00b7160 msgr2=0x7f24b00a8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:38.632 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.630+0000 7f24b6ef9700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24b00b7160 0x7f24b00a8350 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f24a00093f0 tx=0x7f24a0016000 comp rx=0 tx=0).stop 2023-09-30T00:20:38.632 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.630+0000 7f24b6ef9700 1 -- 172.21.15.94:0/2350064286 shutdown_connections 2023-09-30T00:20:38.632 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.630+0000 7f24b6ef9700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f249c098780 0x7f249c09ac30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:38.632 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.630+0000 7f24b6ef9700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24b013fd70 0x7f24b0143140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:38.632 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.630+0000 7f24b6ef9700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24b0070c50 0x7f24b00a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:38.632 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.630+0000 7f24b6ef9700 1 --2- 172.21.15.94:0/2350064286 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24b00b7160 0x7f24b00a8350 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:38.632 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.630+0000 7f24b6ef9700 1 -- 172.21.15.94:0/2350064286 >> 172.21.15.94:0/2350064286 conn(0x7f24b006a090 msgr2=0x7f24b00b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:38.632 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.630+0000 7f24b6ef9700 1 -- 172.21.15.94:0/2350064286 shutdown_connections 2023-09-30T00:20:38.632 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:38.630+0000 7f24b6ef9700 1 -- 172.21.15.94:0/2350064286 wait complete. 2023-09-30T00:20:39.346 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:20:39.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:20:41.644 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:41.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.766+0000 7ff23a686700 1 Processor -- start 2023-09-30T00:20:41.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.766+0000 7ff23a686700 1 -- start start 2023-09-30T00:20:41.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff23a686700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff234070c50 0x7ff2340a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:41.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff23a686700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 0x7ff2341401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:41.770 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff23a686700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2340a1310 con 0x7ff2340b7160 2023-09-30T00:20:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff23a686700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff234071020 con 0x7ff234070c50 2023-09-30T00:20:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff23a686700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2340b76a0 con 0x7ff23413fd70 2023-09-30T00:20:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff238c23700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 0x7ff2341401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff238c23700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 0x7ff2341401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:47900/0 (socket says 172.21.15.94:47900) 2023-09-30T00:20:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff238c23700 1 -- 172.21.15.94:0/2298603922 learned_addr learned my addr 172.21.15.94:0/2298603922 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff238c23700 1 -- 172.21.15.94:0/2298603922 >> v2:172.21.15.121:3300/0 conn(0x7ff234070c50 msgr2=0x7ff2340a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff233fff700 1 --2- 172.21.15.94:0/2298603922 >> v2:172.21.15.121:3300/0 conn(0x7ff234070c50 0x7ff2340a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff238c23700 1 --2- 172.21.15.94:0/2298603922 >> v2:172.21.15.121:3300/0 conn(0x7ff234070c50 0x7ff2340a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff238c23700 1 -- 172.21.15.94:0/2298603922 >> v1:172.21.15.121:6789/0 conn(0x7ff2340b7160 legacy=0x7ff2340a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.767+0000 7ff238c23700 1 -- 172.21.15.94:0/2298603922 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2340b7940 con 0x7ff23413fd70 2023-09-30T00:20:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.768+0000 7ff238c23700 1 --2- 172.21.15.94:0/2298603922 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 0x7ff2341401d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009f20 tx=0x7ff22c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bf0e1d177302b6e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.768+0000 7ff232ffd700 1 -- 172.21.15.94:0/2298603922 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c00e070 con 0x7ff23413fd70 2023-09-30T00:20:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.768+0000 7ff232ffd700 1 -- 172.21.15.94:0/2298603922 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22c00bad0 con 0x7ff23413fd70 2023-09-30T00:20:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.768+0000 7ff232ffd700 1 -- 172.21.15.94:0/2298603922 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0092e0 con 0x7ff23413fd70 2023-09-30T00:20:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.768+0000 7ff23a686700 1 -- 172.21.15.94:0/2298603922 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 msgr2=0x7ff2341401d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:41.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.768+0000 7ff23a686700 1 --2- 172.21.15.94:0/2298603922 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 0x7ff2341401d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009f20 tx=0x7ff22c009840 comp rx=0 tx=0).stop 2023-09-30T00:20:41.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.768+0000 7ff23a686700 1 -- 172.21.15.94:0/2298603922 shutdown_connections 2023-09-30T00:20:41.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.768+0000 7ff23a686700 1 --2- 172.21.15.94:0/2298603922 >> v2:172.21.15.121:3300/0 conn(0x7ff234070c50 0x7ff2340a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:41.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.768+0000 7ff23a686700 1 --2- 172.21.15.94:0/2298603922 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 0x7ff2341401d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:41.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.768+0000 7ff23a686700 1 -- 172.21.15.94:0/2298603922 >> 172.21.15.94:0/2298603922 conn(0x7ff23406a090 msgr2=0x7ff2340702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:41.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.768+0000 7ff23a686700 1 -- 172.21.15.94:0/2298603922 shutdown_connections 2023-09-30T00:20:41.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.768+0000 7ff23a686700 1 -- 172.21.15.94:0/2298603922 wait complete. 2023-09-30T00:20:41.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.769+0000 7ff23a686700 1 Processor -- start 2023-09-30T00:20:41.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff23a686700 1 -- start start 2023-09-30T00:20:41.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff23a686700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff234070c50 0x7ff234142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:41.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff23a686700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2340b7160 0x7ff234144400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff23a686700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 0x7ff234141b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff23a686700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23413c880 con 0x7ff2340b7160 2023-09-30T00:20:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff23a686700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23413ae20 con 0x7ff234070c50 2023-09-30T00:20:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff23a686700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23413b0e0 con 0x7ff23413fd70 2023-09-30T00:20:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff238c23700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 0x7ff234141b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff238c23700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 0x7ff234141b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:47908/0 (socket says 172.21.15.94:47908) 2023-09-30T00:20:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff238c23700 1 -- 172.21.15.94:0/3276037431 learned_addr learned my addr 172.21.15.94:0/3276037431 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff2337fe700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2340b7160 0x7ff234144400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff233fff700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff234070c50 0x7ff234142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff238c23700 1 -- 172.21.15.94:0/3276037431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff234070c50 msgr2=0x7ff234142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff238c23700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff234070c50 0x7ff234142dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff238c23700 1 -- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2340b7160 msgr2=0x7ff234144400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.770+0000 7ff238c23700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2340b7160 0x7ff234144400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.771+0000 7ff238c23700 1 -- 172.21.15.94:0/3276037431 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22c009ae0 con 0x7ff23413fd70 2023-09-30T00:20:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.771+0000 7ff2337fe700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2340b7160 0x7ff234144400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.771+0000 7ff238c23700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 0x7ff234141b20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff2340a97e0 tx=0x7ff22c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.771+0000 7ff21bfff700 1 -- 172.21.15.94:0/3276037431 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c012d30 con 0x7ff23413fd70 2023-09-30T00:20:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.771+0000 7ff21bfff700 1 -- 172.21.15.94:0/3276037431 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22c012e90 con 0x7ff23413fd70 2023-09-30T00:20:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.771+0000 7ff23a686700 1 -- 172.21.15.94:0/3276037431 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff234143300 con 0x7ff23413fd70 2023-09-30T00:20:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.771+0000 7ff21bfff700 1 -- 172.21.15.94:0/3276037431 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c01d670 con 0x7ff23413fd70 2023-09-30T00:20:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.771+0000 7ff23a686700 1 -- 172.21.15.94:0/3276037431 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff23413c630 con 0x7ff23413fd70 2023-09-30T00:20:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.772+0000 7ff23a686700 1 -- 172.21.15.94:0/3276037431 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff234144100 con 0x7ff23413fd70 2023-09-30T00:20:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.773+0000 7ff21bfff700 1 -- 172.21.15.94:0/3276037431 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ff22c01da60 con 0x7ff23413fd70 2023-09-30T00:20:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.773+0000 7ff21bfff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.773+0000 7ff21bfff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.773+0000 7ff21bfff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.773+0000 7ff21bfff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.773+0000 7ff21bfff700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff220098780 0x7ff22009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.773+0000 7ff233fff700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff220098780 0x7ff22009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.773+0000 7ff21bfff700 1 -- 172.21.15.94:0/3276037431 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7ff22c0c20b0 con 0x7ff23413fd70 2023-09-30T00:20:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.774+0000 7ff233fff700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff220098780 0x7ff22009ac30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff22400ab30 tx=0x7ff224008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:41.780 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.777+0000 7ff21bfff700 1 -- 172.21.15.94:0/3276037431 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff22c08f780 con 0x7ff23413fd70 2023-09-30T00:20:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.998+0000 7ff23a686700 1 -- 172.21.15.94:0/3276037431 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff2340a1310 con 0x7ff23413fd70 2023-09-30T00:20:42.000 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:41.998+0000 7ff21bfff700 1 -- 172.21.15.94:0/3276037431 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff2340a1310 con 0x7ff23413fd70 2023-09-30T00:20:42.000 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:42.000 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:42.002 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.000+0000 7ff23a686700 10 mgrc shutdown 2023-09-30T00:20:42.002 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.000+0000 7ff23a686700 1 -- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff220098780 msgr2=0x7ff22009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:42.002 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.000+0000 7ff23a686700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff220098780 0x7ff22009ac30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff22400ab30 tx=0x7ff224008040 comp rx=0 tx=0).stop 2023-09-30T00:20:42.002 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.000+0000 7ff23a686700 1 -- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 msgr2=0x7ff234141b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.000+0000 7ff23a686700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 0x7ff234141b20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff2340a97e0 tx=0x7ff22c017000 comp rx=0 tx=0).stop 2023-09-30T00:20:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.001+0000 7ff23a686700 1 -- 172.21.15.94:0/3276037431 shutdown_connections 2023-09-30T00:20:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.001+0000 7ff23a686700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff220098780 0x7ff22009ac30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.001+0000 7ff23a686700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff23413fd70 0x7ff234141b20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.001+0000 7ff23a686700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff234070c50 0x7ff234142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.001+0000 7ff23a686700 1 --2- 172.21.15.94:0/3276037431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2340b7160 0x7ff234144400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.001+0000 7ff23a686700 1 -- 172.21.15.94:0/3276037431 >> 172.21.15.94:0/3276037431 conn(0x7ff23406a090 msgr2=0x7ff23413cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.001+0000 7ff23a686700 1 -- 172.21.15.94:0/3276037431 shutdown_connections 2023-09-30T00:20:42.003 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:42.001+0000 7ff23a686700 1 -- 172.21.15.94:0/3276037431 wait complete. 2023-09-30T00:20:45.014 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:45.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.134+0000 7f6a5a996700 1 Processor -- start 2023-09-30T00:20:45.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.134+0000 7f6a5a996700 1 -- start start 2023-09-30T00:20:45.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a5a996700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6a54070c50 0x7f6a540a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:45.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a5a996700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a540a1310 con 0x7f6a540b7160 2023-09-30T00:20:45.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a5a996700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a54071020 con 0x7f6a5413fd70 2023-09-30T00:20:45.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a5a996700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a540b76a0 con 0x7f6a54070c50 2023-09-30T00:20:45.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a53fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6a54070c50 0x7f6a540a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:45.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a537fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6a540b7160 0x7f6a540a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35104/0 (socket says 172.21.15.94:35104) 2023-09-30T00:20:45.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a537fe700 1 -- 172.21.15.94:0/1910263059 learned_addr learned my addr 172.21.15.94:0/1910263059 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:20:45.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a52ffd700 1 -- 172.21.15.94:0/1910263059 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254023475 0 0) 0x7f6a48001110 con 0x7f6a540b7160 2023-09-30T00:20:45.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a52ffd700 1 -- 172.21.15.94:0/1910263059 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a3c003570 con 0x7f6a540b7160 2023-09-30T00:20:45.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a53fff700 1 -- 172.21.15.94:0/1910263059 >> v1:172.21.15.121:6789/0 conn(0x7f6a5413fd70 legacy=0x7f6a5414b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:45.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a53fff700 1 -- 172.21.15.94:0/1910263059 >> v1:172.21.15.94:6789/0 conn(0x7f6a540b7160 legacy=0x7f6a540a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:45.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a53fff700 1 -- 172.21.15.94:0/1910263059 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a540b7940 con 0x7f6a54070c50 2023-09-30T00:20:45.138 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.135+0000 7f6a53fff700 1 --2- 172.21.15.94:0/1910263059 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6a54070c50 0x7f6a540a0dd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6a44009910 tx=0x7f6a440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2d1ab207b2fdd82 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:45.139 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.136+0000 7f6a52ffd700 1 -- 172.21.15.94:0/1910263059 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4400b710 con 0x7f6a54070c50 2023-09-30T00:20:45.139 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.136+0000 7f6a52ffd700 1 -- 172.21.15.94:0/1910263059 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4400b870 con 0x7f6a54070c50 2023-09-30T00:20:45.139 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.136+0000 7f6a52ffd700 1 -- 172.21.15.94:0/1910263059 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4400bbf0 con 0x7f6a54070c50 2023-09-30T00:20:45.141 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.136+0000 7f6a5a996700 1 -- 172.21.15.94:0/1910263059 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6a54070c50 msgr2=0x7f6a540a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:45.141 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.136+0000 7f6a5a996700 1 --2- 172.21.15.94:0/1910263059 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6a54070c50 0x7f6a540a0dd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6a44009910 tx=0x7f6a440091a0 comp rx=0 tx=0).stop 2023-09-30T00:20:45.141 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.136+0000 7f6a5a996700 1 -- 172.21.15.94:0/1910263059 shutdown_connections 2023-09-30T00:20:45.141 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.136+0000 7f6a5a996700 1 --2- 172.21.15.94:0/1910263059 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6a54070c50 0x7f6a540a0dd0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:45.142 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.136+0000 7f6a5a996700 1 -- 172.21.15.94:0/1910263059 >> 172.21.15.94:0/1910263059 conn(0x7f6a5406a090 msgr2=0x7f6a540702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:45.142 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.136+0000 7f6a5a996700 1 -- 172.21.15.94:0/1910263059 shutdown_connections 2023-09-30T00:20:45.142 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.136+0000 7f6a5a996700 1 -- 172.21.15.94:0/1910263059 wait complete. 2023-09-30T00:20:45.142 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.137+0000 7f6a5a996700 1 Processor -- start 2023-09-30T00:20:45.142 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.137+0000 7f6a5a996700 1 -- start start 2023-09-30T00:20:45.142 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a5a996700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6a54070c50 0x7f6a540a6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:45.142 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a5a996700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a540b7160 0x7f6a540adec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:45.142 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a5a996700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a5413fd70 0x7f6a540a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:45.142 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a5a996700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a540ada50 con 0x7f6a540b7160 2023-09-30T00:20:45.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a5a996700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a54151c10 con 0x7f6a5413fd70 2023-09-30T00:20:45.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a5a996700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a54150f50 con 0x7f6a54070c50 2023-09-30T00:20:45.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a537fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a540b7160 0x7f6a540adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:45.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a537fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a540b7160 0x7f6a540adec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43700/0 (socket says 172.21.15.94:43700) 2023-09-30T00:20:45.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a537fe700 1 -- 172.21.15.94:0/1276658109 learned_addr learned my addr 172.21.15.94:0/1276658109 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:45.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a53fff700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6a54070c50 0x7f6a540a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:45.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a58f33700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a5413fd70 0x7f6a540a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:45.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a537fe700 1 -- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6a54070c50 msgr2=0x7f6a540a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:45.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a537fe700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6a54070c50 0x7f6a540a6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:45.143 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a537fe700 1 -- 172.21.15.94:0/1276658109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a5413fd70 msgr2=0x7f6a540a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:45.144 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a537fe700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a5413fd70 0x7f6a540a5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:45.144 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a537fe700 1 -- 172.21.15.94:0/1276658109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a44009420 con 0x7f6a540b7160 2023-09-30T00:20:45.144 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a537fe700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a540b7160 0x7f6a540adec0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800c7f0 tx=0x7f6a48007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:45.144 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.138+0000 7f6a3bfff700 1 -- 172.21.15.94:0/1276658109 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48005b30 con 0x7f6a540b7160 2023-09-30T00:20:45.144 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.139+0000 7f6a3bfff700 1 -- 172.21.15.94:0/1276658109 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a48005cb0 con 0x7f6a540b7160 2023-09-30T00:20:45.144 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.139+0000 7f6a5a996700 1 -- 172.21.15.94:0/1276658109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a54151210 con 0x7f6a540b7160 2023-09-30T00:20:45.144 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.139+0000 7f6a3bfff700 1 -- 172.21.15.94:0/1276658109 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48007320 con 0x7f6a540b7160 2023-09-30T00:20:45.144 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.139+0000 7f6a5a996700 1 -- 172.21.15.94:0/1276658109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a540a5bb0 con 0x7f6a540b7160 2023-09-30T00:20:45.144 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.140+0000 7f6a5a996700 1 -- 172.21.15.94:0/1276658109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a54071630 con 0x7f6a540b7160 2023-09-30T00:20:45.145 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.141+0000 7f6a3bfff700 1 -- 172.21.15.94:0/1276658109 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6a4801b3b0 con 0x7f6a540b7160 2023-09-30T00:20:45.145 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.141+0000 7f6a3bfff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:45.145 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.142+0000 7f6a3bfff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:45.145 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.142+0000 7f6a3bfff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:45.145 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.142+0000 7f6a3bfff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:45.146 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.142+0000 7f6a3bfff700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6a40098780 0x7f6a4009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:45.146 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.142+0000 7f6a53fff700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6a40098780 0x7f6a4009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:45.146 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.142+0000 7f6a3bfff700 1 -- 172.21.15.94:0/1276658109 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f6a480bfc90 con 0x7f6a540b7160 2023-09-30T00:20:45.146 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.142+0000 7f6a53fff700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6a40098780 0x7f6a4009ac30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6a440091e0 tx=0x7f6a44017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:45.146 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.145+0000 7f6a3bfff700 1 -- 172.21.15.94:0/1276658109 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6a4808d360 con 0x7f6a540b7160 2023-09-30T00:20:45.372 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.371+0000 7f6a5a996700 1 -- 172.21.15.94:0/1276658109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6a540a1310 con 0x7f6a540b7160 2023-09-30T00:20:45.372 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.371+0000 7f6a3bfff700 1 -- 172.21.15.94:0/1276658109 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6a48017100 con 0x7f6a540b7160 2023-09-30T00:20:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:45.373 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:45.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 10 mgrc shutdown 2023-09-30T00:20:45.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 1 -- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6a40098780 msgr2=0x7f6a4009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:45.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6a40098780 0x7f6a4009ac30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6a440091e0 tx=0x7f6a44017000 comp rx=0 tx=0).stop 2023-09-30T00:20:45.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 1 -- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a540b7160 msgr2=0x7f6a540adec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:45.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a540b7160 0x7f6a540adec0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800c7f0 tx=0x7f6a48007c70 comp rx=0 tx=0).stop 2023-09-30T00:20:45.375 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 1 -- 172.21.15.94:0/1276658109 shutdown_connections 2023-09-30T00:20:45.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6a40098780 0x7f6a4009ac30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:45.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6a54070c50 0x7f6a540a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:45.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a5413fd70 0x7f6a540a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:45.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 1 --2- 172.21.15.94:0/1276658109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a540b7160 0x7f6a540adec0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:45.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 1 -- 172.21.15.94:0/1276658109 >> 172.21.15.94:0/1276658109 conn(0x7f6a5406a090 msgr2=0x7f6a5413d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:45.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 1 -- 172.21.15.94:0/1276658109 shutdown_connections 2023-09-30T00:20:45.376 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:45.374+0000 7f6a5a996700 1 -- 172.21.15.94:0/1276658109 wait complete. 2023-09-30T00:20:48.386 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:48.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.511+0000 7f346727f700 1 Processor -- start 2023-09-30T00:20:48.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.511+0000 7f346727f700 1 -- start start 2023-09-30T00:20:48.514 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f346727f700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f34600a0dd0 0x7f346014c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:48.514 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f346727f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3460071020 con 0x7f34600b7160 2023-09-30T00:20:48.514 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f346727f700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34600b76a0 con 0x7f3460070c50 2023-09-30T00:20:48.514 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f346727f700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34600b7940 con 0x7f34600a0dd0 2023-09-30T00:20:48.514 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f346581c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f34600a0dd0 0x7f346014c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:48.514 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f346481a700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f34600b7160 0x7f346013fd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35106/0 (socket says 172.21.15.94:35106) 2023-09-30T00:20:48.514 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f346481a700 1 -- 172.21.15.94:0/4069408701 learned_addr learned my addr 172.21.15.94:0/4069408701 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:20:48.514 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f345ffff700 1 -- 172.21.15.94:0/4069408701 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537249697 0 0) 0x7f3460071020 con 0x7f34600b7160 2023-09-30T00:20:48.514 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f345ffff700 1 -- 172.21.15.94:0/4069408701 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3448003570 con 0x7f34600b7160 2023-09-30T00:20:48.515 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f346581c700 1 -- 172.21.15.94:0/4069408701 >> v1:172.21.15.121:6789/0 conn(0x7f3460070c50 legacy=0x7f34600a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:48.515 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f346581c700 1 -- 172.21.15.94:0/4069408701 >> v1:172.21.15.94:6789/0 conn(0x7f34600b7160 legacy=0x7f346013fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:48.515 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f346581c700 1 -- 172.21.15.94:0/4069408701 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34600a2070 con 0x7f34600a0dd0 2023-09-30T00:20:48.515 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f346581c700 1 --2- 172.21.15.94:0/4069408701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600a0dd0 0x7f346014c0d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3458009e60 tx=0x7f3458009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f57c88afb478b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:48.515 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f345ffff700 1 -- 172.21.15.94:0/4069408701 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345800e070 con 0x7f34600a0dd0 2023-09-30T00:20:48.515 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f345ffff700 1 -- 172.21.15.94:0/4069408701 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345800bbf0 con 0x7f34600a0dd0 2023-09-30T00:20:48.515 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.512+0000 7f345ffff700 1 -- 172.21.15.94:0/4069408701 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458012430 con 0x7f34600a0dd0 2023-09-30T00:20:48.515 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.513+0000 7f346727f700 1 -- 172.21.15.94:0/4069408701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600a0dd0 msgr2=0x7f346014c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:48.515 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.513+0000 7f346727f700 1 --2- 172.21.15.94:0/4069408701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600a0dd0 0x7f346014c0d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3458009e60 tx=0x7f3458009780 comp rx=0 tx=0).stop 2023-09-30T00:20:48.515 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.513+0000 7f346727f700 1 -- 172.21.15.94:0/4069408701 shutdown_connections 2023-09-30T00:20:48.516 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.513+0000 7f346727f700 1 --2- 172.21.15.94:0/4069408701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600a0dd0 0x7f346014c0d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:48.516 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.513+0000 7f346727f700 1 -- 172.21.15.94:0/4069408701 >> 172.21.15.94:0/4069408701 conn(0x7f346006a090 msgr2=0x7f34600702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:48.516 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.513+0000 7f346727f700 1 -- 172.21.15.94:0/4069408701 shutdown_connections 2023-09-30T00:20:48.516 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.513+0000 7f346727f700 1 -- 172.21.15.94:0/4069408701 wait complete. 2023-09-30T00:20:48.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.514+0000 7f346727f700 1 Processor -- start 2023-09-30T00:20:48.519 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.514+0000 7f346727f700 1 -- start start 2023-09-30T00:20:48.519 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346727f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3460070c50 0x7f346012dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:48.519 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346727f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34600a0dd0 0x7f3460142df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:48.519 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346727f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34600b7160 0x7f34601504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:48.519 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346727f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346012e920 con 0x7f3460070c50 2023-09-30T00:20:48.519 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346727f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346012eb20 con 0x7f34600a0dd0 2023-09-30T00:20:48.519 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346727f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3460150fd0 con 0x7f34600b7160 2023-09-30T00:20:48.519 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346501b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3460070c50 0x7f346012dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:48.519 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346501b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3460070c50 0x7f346012dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43726/0 (socket says 172.21.15.94:43726) 2023-09-30T00:20:48.520 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346501b700 1 -- 172.21.15.94:0/3377329243 learned_addr learned my addr 172.21.15.94:0/3377329243 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:48.520 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346581c700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34600b7160 0x7f34601504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:48.520 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346481a700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34600a0dd0 0x7f3460142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:48.520 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346501b700 1 -- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34600b7160 msgr2=0x7f34601504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:48.520 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346501b700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34600b7160 0x7f34601504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:48.520 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346501b700 1 -- 172.21.15.94:0/3377329243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34600a0dd0 msgr2=0x7f3460142df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:48.520 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346501b700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34600a0dd0 0x7f3460142df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:48.520 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346501b700 1 -- 172.21.15.94:0/3377329243 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3458009a20 con 0x7f3460070c50 2023-09-30T00:20:48.520 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346501b700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3460070c50 0x7f346012dfe0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f345000c590 tx=0x7f345000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:48.520 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f345d7fa700 1 -- 172.21.15.94:0/3377329243 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450011590 con 0x7f3460070c50 2023-09-30T00:20:48.520 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f345d7fa700 1 -- 172.21.15.94:0/3377329243 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3450011710 con 0x7f3460070c50 2023-09-30T00:20:48.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f346727f700 1 -- 172.21.15.94:0/3377329243 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3460151290 con 0x7f3460070c50 2023-09-30T00:20:48.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.515+0000 7f345d7fa700 1 -- 172.21.15.94:0/3377329243 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450011ad0 con 0x7f3460070c50 2023-09-30T00:20:48.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.516+0000 7f346727f700 1 -- 172.21.15.94:0/3377329243 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3460143330 con 0x7f3460070c50 2023-09-30T00:20:48.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.516+0000 7f346727f700 1 -- 172.21.15.94:0/3377329243 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3460070840 con 0x7f3460070c50 2023-09-30T00:20:48.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.517+0000 7f345d7fa700 1 -- 172.21.15.94:0/3377329243 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f345001b3b0 con 0x7f3460070c50 2023-09-30T00:20:48.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.517+0000 7f345d7fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:48.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.518+0000 7f345d7fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:48.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.518+0000 7f345d7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:48.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.518+0000 7f345d7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:48.522 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.518+0000 7f345d7fa700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f344c098780 0x7f344c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:48.522 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.518+0000 7f346481a700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f344c098780 0x7f344c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:48.522 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.518+0000 7f345d7fa700 1 -- 172.21.15.94:0/3377329243 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f34500bfbc0 con 0x7f3460070c50 2023-09-30T00:20:48.522 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.518+0000 7f346481a700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f344c098780 0x7f344c09ac30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3460143b40 tx=0x7f3454009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:48.522 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.521+0000 7f345d7fa700 1 -- 172.21.15.94:0/3377329243 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f345008d290 con 0x7f3460070c50 2023-09-30T00:20:48.739 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.739+0000 7f346727f700 1 -- 172.21.15.94:0/3377329243 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3460071630 con 0x7f3460070c50 2023-09-30T00:20:48.740 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.740+0000 7f345d7fa700 1 -- 172.21.15.94:0/3377329243 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3450017080 con 0x7f3460070c50 2023-09-30T00:20:48.740 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:48.740 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:48.742 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.741+0000 7f346727f700 10 mgrc shutdown 2023-09-30T00:20:48.742 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.741+0000 7f346727f700 1 -- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f344c098780 msgr2=0x7f344c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:48.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.742+0000 7f346727f700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f344c098780 0x7f344c09ac30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3460143b40 tx=0x7f3454009040 comp rx=0 tx=0).stop 2023-09-30T00:20:48.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.742+0000 7f346727f700 1 -- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3460070c50 msgr2=0x7f346012dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:48.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.742+0000 7f346727f700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3460070c50 0x7f346012dfe0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f345000c590 tx=0x7f345000c8a0 comp rx=0 tx=0).stop 2023-09-30T00:20:48.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.742+0000 7f346727f700 1 -- 172.21.15.94:0/3377329243 shutdown_connections 2023-09-30T00:20:48.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.742+0000 7f346727f700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f344c098780 0x7f344c09ac30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:48.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.742+0000 7f346727f700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34600b7160 0x7f34601504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:48.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.742+0000 7f346727f700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3460070c50 0x7f346012dfe0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:48.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.742+0000 7f346727f700 1 --2- 172.21.15.94:0/3377329243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34600a0dd0 0x7f3460142df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:48.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.742+0000 7f346727f700 1 -- 172.21.15.94:0/3377329243 >> 172.21.15.94:0/3377329243 conn(0x7f346006a090 msgr2=0x7f346013fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:48.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.742+0000 7f346727f700 1 -- 172.21.15.94:0/3377329243 shutdown_connections 2023-09-30T00:20:48.744 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:48.742+0000 7f346727f700 1 -- 172.21.15.94:0/3377329243 wait complete. 2023-09-30T00:20:51.755 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:51.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.876+0000 7f73d397b700 1 Processor -- start 2023-09-30T00:20:51.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.876+0000 7f73d397b700 1 -- start start 2023-09-30T00:20:51.879 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.877+0000 7f73d397b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f73cc0a46c0 0x7f73cc068c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:51.879 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.877+0000 7f73d397b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73cc0b5850 0x7f73cc14b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:51.879 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.877+0000 7f73d397b700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73cc069760 con 0x7f73cc06a460 2023-09-30T00:20:51.879 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.877+0000 7f73d397b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73cc0a57f0 con 0x7f73cc0b5850 2023-09-30T00:20:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.877+0000 7f73d397b700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73cc0a4a90 con 0x7f73cc0a46c0 2023-09-30T00:20:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.877+0000 7f73d1f18700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73cc0b5850 0x7f73cc14b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.877+0000 7f73d1f18700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73cc0b5850 0x7f73cc14b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43742/0 (socket says 172.21.15.94:43742) 2023-09-30T00:20:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.877+0000 7f73d1f18700 1 -- 172.21.15.94:0/1480488630 learned_addr learned my addr 172.21.15.94:0/1480488630 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.877+0000 7f73d1717700 1 --2- 172.21.15.94:0/1480488630 >> v2:172.21.15.121:3300/0 conn(0x7f73cc0a46c0 0x7f73cc068c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73d1f18700 1 -- 172.21.15.94:0/1480488630 >> v2:172.21.15.121:3300/0 conn(0x7f73cc0a46c0 msgr2=0x7f73cc068c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73d1f18700 1 --2- 172.21.15.94:0/1480488630 >> v2:172.21.15.121:3300/0 conn(0x7f73cc0a46c0 0x7f73cc068c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73d1f18700 1 -- 172.21.15.94:0/1480488630 >> v1:172.21.15.121:6789/0 conn(0x7f73cc06a460 legacy=0x7f73cc0b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73d1f18700 1 -- 172.21.15.94:0/1480488630 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc069c80 con 0x7f73cc0b5850 2023-09-30T00:20:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73d1f18700 1 --2- 172.21.15.94:0/1480488630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73cc0b5850 0x7f73cc14b060 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f73c40099b0 tx=0x7f73c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4018ab5926d15b12 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73c3fff700 1 -- 172.21.15.94:0/1480488630 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c400b820 con 0x7f73cc0b5850 2023-09-30T00:20:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73c3fff700 1 -- 172.21.15.94:0/1480488630 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c400b980 con 0x7f73cc0b5850 2023-09-30T00:20:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73c3fff700 1 -- 172.21.15.94:0/1480488630 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c400bd20 con 0x7f73cc0b5850 2023-09-30T00:20:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73d397b700 1 -- 172.21.15.94:0/1480488630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73cc0b5850 msgr2=0x7f73cc14b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73d397b700 1 --2- 172.21.15.94:0/1480488630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73cc0b5850 0x7f73cc14b060 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f73c40099b0 tx=0x7f73c4009250 comp rx=0 tx=0).stop 2023-09-30T00:20:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73d397b700 1 -- 172.21.15.94:0/1480488630 shutdown_connections 2023-09-30T00:20:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73d397b700 1 --2- 172.21.15.94:0/1480488630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73cc0b5850 0x7f73cc14b060 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73d397b700 1 --2- 172.21.15.94:0/1480488630 >> v2:172.21.15.121:3300/0 conn(0x7f73cc0a46c0 0x7f73cc068c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.878+0000 7f73d397b700 1 -- 172.21.15.94:0/1480488630 >> 172.21.15.94:0/1480488630 conn(0x7f73cc06cbf0 msgr2=0x7f73cc06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.879+0000 7f73d397b700 1 -- 172.21.15.94:0/1480488630 shutdown_connections 2023-09-30T00:20:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.879+0000 7f73d397b700 1 -- 172.21.15.94:0/1480488630 wait complete. 2023-09-30T00:20:51.884 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.879+0000 7f73d397b700 1 Processor -- start 2023-09-30T00:20:51.884 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.879+0000 7f73d397b700 1 -- start start 2023-09-30T00:20:51.884 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.880+0000 7f73d397b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73cc06a460 0x7f73cc0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:51.884 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.880+0000 7f73d397b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73cc0a46c0 0x7f73cc11ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:51.884 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.880+0000 7f73d397b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73cc0b5850 0x7f73cc0af850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:51.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.880+0000 7f73d397b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0af3c0 con 0x7f73cc0a46c0 2023-09-30T00:20:51.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.880+0000 7f73d397b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc11d3c0 con 0x7f73cc0b5850 2023-09-30T00:20:51.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.880+0000 7f73d397b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0b04e0 con 0x7f73cc06a460 2023-09-30T00:20:51.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.880+0000 7f73d1717700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73cc06a460 0x7f73cc0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:51.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.880+0000 7f73d1717700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73cc06a460 0x7f73cc0b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:43996/0 (socket says 172.21.15.94:43996) 2023-09-30T00:20:51.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.880+0000 7f73d1717700 1 -- 172.21.15.94:0/931403589 learned_addr learned my addr 172.21.15.94:0/931403589 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:51.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.880+0000 7f73d0f16700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73cc0a46c0 0x7f73cc11ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:51.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.880+0000 7f73d1f18700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73cc0b5850 0x7f73cc0af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:51.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.881+0000 7f73d0f16700 1 -- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73cc06a460 msgr2=0x7f73cc0b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:51.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.881+0000 7f73d0f16700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73cc06a460 0x7f73cc0b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:51.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.881+0000 7f73d0f16700 1 -- 172.21.15.94:0/931403589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73cc0b5850 msgr2=0x7f73cc0af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:51.885 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.881+0000 7f73d0f16700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73cc0b5850 0x7f73cc0af850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:51.886 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.881+0000 7f73d0f16700 1 -- 172.21.15.94:0/931403589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c40094f0 con 0x7f73cc0a46c0 2023-09-30T00:20:51.886 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.881+0000 7f73d1717700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73cc06a460 0x7f73cc0b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:51.886 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.881+0000 7f73d0f16700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73cc0a46c0 0x7f73cc11ce80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00c5c0 tx=0x7f73bc007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:51.886 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.881+0000 7f73c17fa700 1 -- 172.21.15.94:0/931403589 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc011590 con 0x7f73cc0a46c0 2023-09-30T00:20:51.886 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.881+0000 7f73c17fa700 1 -- 172.21.15.94:0/931403589 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73bc011710 con 0x7f73cc0a46c0 2023-09-30T00:20:51.886 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.881+0000 7f73d397b700 1 -- 172.21.15.94:0/931403589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73cc0ae680 con 0x7f73cc0a46c0 2023-09-30T00:20:51.886 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.881+0000 7f73c17fa700 1 -- 172.21.15.94:0/931403589 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc011ad0 con 0x7f73cc0a46c0 2023-09-30T00:20:51.886 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.881+0000 7f73d397b700 1 -- 172.21.15.94:0/931403589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73cc0affd0 con 0x7f73cc0a46c0 2023-09-30T00:20:51.886 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.882+0000 7f73d397b700 1 -- 172.21.15.94:0/931403589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73cc0af570 con 0x7f73cc0a46c0 2023-09-30T00:20:51.886 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.883+0000 7f73c17fa700 1 -- 172.21.15.94:0/931403589 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f73bc01b3b0 con 0x7f73cc0a46c0 2023-09-30T00:20:51.886 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.883+0000 7f73c17fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:51.887 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.883+0000 7f73c17fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:51.887 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.883+0000 7f73c17fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:51.887 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.883+0000 7f73c17fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:51.887 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.883+0000 7f73c17fa700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f73b0098780 0x7f73b009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:51.887 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.883+0000 7f73d1717700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f73b0098780 0x7f73b009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:51.887 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.883+0000 7f73c17fa700 1 -- 172.21.15.94:0/931403589 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f73bc0bfc00 con 0x7f73cc0a46c0 2023-09-30T00:20:51.887 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.884+0000 7f73d1717700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f73b0098780 0x7f73b009ac30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f73c8007a90 tx=0x7f73c8009900 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:51.887 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:51.887+0000 7f73c17fa700 1 -- 172.21.15.94:0/931403589 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73bc08d1a0 con 0x7f73cc0a46c0 2023-09-30T00:20:52.111 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.111+0000 7f73d397b700 1 -- 172.21.15.94:0/931403589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f73cc0af570 con 0x7f73cc0a46c0 2023-09-30T00:20:52.112 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.111+0000 7f73c17fa700 1 -- 172.21.15.94:0/931403589 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f73cc0af570 con 0x7f73cc0a46c0 2023-09-30T00:20:52.112 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:52.112 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:52.114 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 10 mgrc shutdown 2023-09-30T00:20:52.114 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 1 -- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f73b0098780 msgr2=0x7f73b009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:52.114 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f73b0098780 0x7f73b009ac30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f73c8007a90 tx=0x7f73c8009900 comp rx=0 tx=0).stop 2023-09-30T00:20:52.114 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 1 -- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73cc0a46c0 msgr2=0x7f73cc11ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:52.114 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73cc0a46c0 0x7f73cc11ce80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00c5c0 tx=0x7f73bc007ec0 comp rx=0 tx=0).stop 2023-09-30T00:20:52.114 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 1 -- 172.21.15.94:0/931403589 shutdown_connections 2023-09-30T00:20:52.114 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f73b0098780 0x7f73b009ac30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:52.114 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73cc06a460 0x7f73cc0b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:52.115 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73cc0b5850 0x7f73cc0af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:52.115 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 1 --2- 172.21.15.94:0/931403589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73cc0a46c0 0x7f73cc11ce80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:52.115 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 1 -- 172.21.15.94:0/931403589 >> 172.21.15.94:0/931403589 conn(0x7f73cc06cbf0 msgr2=0x7f73cc0b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:52.115 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 1 -- 172.21.15.94:0/931403589 shutdown_connections 2023-09-30T00:20:52.115 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:52.113+0000 7f73d397b700 1 -- 172.21.15.94:0/931403589 wait complete. 2023-09-30T00:20:55.126 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:55.245 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.245+0000 7fc0cfad6700 1 Processor -- start 2023-09-30T00:20:55.245 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.245+0000 7fc0cfad6700 1 -- start start 2023-09-30T00:20:55.247 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.245+0000 7fc0cfad6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 0x7fc0c80a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:55.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cfad6700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc0c813fd70 0x7fc0c81401f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:55.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cfad6700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c80a1310 con 0x7fc0c80b7160 2023-09-30T00:20:55.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cfad6700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c8071020 con 0x7fc0c813fd70 2023-09-30T00:20:55.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cfad6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c80b76a0 con 0x7fc0c8070c50 2023-09-30T00:20:55.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cd872700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 0x7fc0c80a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:55.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cd872700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 0x7fc0c80a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:33262/0 (socket says 172.21.15.94:33262) 2023-09-30T00:20:55.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cd872700 1 -- 172.21.15.94:0/74855025 learned_addr learned my addr 172.21.15.94:0/74855025 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:55.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cd872700 1 -- 172.21.15.94:0/74855025 >> v2:172.21.15.121:3300/0 conn(0x7fc0c813fd70 msgr2=0x7fc0c81401f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:55.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cd872700 1 --2- 172.21.15.94:0/74855025 >> v2:172.21.15.121:3300/0 conn(0x7fc0c813fd70 0x7fc0c81401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:55.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cd872700 1 -- 172.21.15.94:0/74855025 >> v1:172.21.15.121:6789/0 conn(0x7fc0c80b7160 legacy=0x7fc0c80a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:55.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cd872700 1 -- 172.21.15.94:0/74855025 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c80b7940 con 0x7fc0c8070c50 2023-09-30T00:20:55.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cd872700 1 --2- 172.21.15.94:0/74855025 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 0x7fc0c80a0dd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc0c40099b0 tx=0x7fc0c4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a78a7ab64afa411 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:55.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cc870700 1 -- 172.21.15.94:0/74855025 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c400b8b0 con 0x7fc0c8070c50 2023-09-30T00:20:55.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cc870700 1 -- 172.21.15.94:0/74855025 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c400ba10 con 0x7fc0c8070c50 2023-09-30T00:20:55.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.246+0000 7fc0cc870700 1 -- 172.21.15.94:0/74855025 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c400bdb0 con 0x7fc0c8070c50 2023-09-30T00:20:55.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.247+0000 7fc0cfad6700 1 -- 172.21.15.94:0/74855025 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 msgr2=0x7fc0c80a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:55.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.247+0000 7fc0cfad6700 1 --2- 172.21.15.94:0/74855025 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 0x7fc0c80a0dd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc0c40099b0 tx=0x7fc0c4009250 comp rx=0 tx=0).stop 2023-09-30T00:20:55.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.247+0000 7fc0cfad6700 1 -- 172.21.15.94:0/74855025 shutdown_connections 2023-09-30T00:20:55.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.247+0000 7fc0cfad6700 1 --2- 172.21.15.94:0/74855025 >> v2:172.21.15.121:3300/0 conn(0x7fc0c813fd70 0x7fc0c81401f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:55.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.247+0000 7fc0cfad6700 1 --2- 172.21.15.94:0/74855025 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 0x7fc0c80a0dd0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:55.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.247+0000 7fc0cfad6700 1 -- 172.21.15.94:0/74855025 >> 172.21.15.94:0/74855025 conn(0x7fc0c806a090 msgr2=0x7fc0c80702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:55.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.247+0000 7fc0cfad6700 1 -- 172.21.15.94:0/74855025 shutdown_connections 2023-09-30T00:20:55.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.247+0000 7fc0cfad6700 1 -- 172.21.15.94:0/74855025 wait complete. 2023-09-30T00:20:55.252 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.248+0000 7fc0cfad6700 1 Processor -- start 2023-09-30T00:20:55.252 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.248+0000 7fc0cfad6700 1 -- start start 2023-09-30T00:20:55.252 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.248+0000 7fc0cfad6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 0x7fc0c8141b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:55.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.248+0000 7fc0cfad6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0c80b7160 0x7fc0c809f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:55.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.248+0000 7fc0cfad6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0c813fd70 0x7fc0c813a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:55.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.248+0000 7fc0cfad6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c813dad0 con 0x7fc0c80b7160 2023-09-30T00:20:55.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.248+0000 7fc0cfad6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c813acb0 con 0x7fc0c813fd70 2023-09-30T00:20:55.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.248+0000 7fc0cfad6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c813af70 con 0x7fc0c8070c50 2023-09-30T00:20:55.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.248+0000 7fc0cd872700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 0x7fc0c8141b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:55.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.248+0000 7fc0cd872700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 0x7fc0c8141b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:33270/0 (socket says 172.21.15.94:33270) 2023-09-30T00:20:55.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.248+0000 7fc0cd872700 1 -- 172.21.15.94:0/1561197088 learned_addr learned my addr 172.21.15.94:0/1561197088 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:55.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.248+0000 7fc0cd071700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0c80b7160 0x7fc0c809f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:55.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.249+0000 7fc0ce073700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0c813fd70 0x7fc0c813a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:55.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.249+0000 7fc0cd872700 1 -- 172.21.15.94:0/1561197088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0c813fd70 msgr2=0x7fc0c813a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:55.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.249+0000 7fc0cd872700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0c813fd70 0x7fc0c813a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:55.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.249+0000 7fc0cd872700 1 -- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0c80b7160 msgr2=0x7fc0c809f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:55.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.249+0000 7fc0cd872700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0c80b7160 0x7fc0c809f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:55.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.249+0000 7fc0cd872700 1 -- 172.21.15.94:0/1561197088 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c40094f0 con 0x7fc0c8070c50 2023-09-30T00:20:55.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.249+0000 7fc0cd872700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 0x7fc0c8141b20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009290 tx=0x7fc0c4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:55.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.249+0000 7fc0bdffb700 1 -- 172.21.15.94:0/1561197088 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4026480 con 0x7fc0c8070c50 2023-09-30T00:20:55.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.249+0000 7fc0bdffb700 1 -- 172.21.15.94:0/1561197088 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c401f070 con 0x7fc0c8070c50 2023-09-30T00:20:55.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.249+0000 7fc0bdffb700 1 -- 172.21.15.94:0/1561197088 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4026870 con 0x7fc0c8070c50 2023-09-30T00:20:55.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.249+0000 7fc0cfad6700 1 -- 172.21.15.94:0/1561197088 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c813b8b0 con 0x7fc0c8070c50 2023-09-30T00:20:55.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.249+0000 7fc0cfad6700 1 -- 172.21.15.94:0/1561197088 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c809bdd0 con 0x7fc0c8070c50 2023-09-30T00:20:55.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.250+0000 7fc0cfad6700 1 -- 172.21.15.94:0/1561197088 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c8071630 con 0x7fc0c8070c50 2023-09-30T00:20:55.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.251+0000 7fc0bdffb700 1 -- 172.21.15.94:0/1561197088 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc0c401a440 con 0x7fc0c8070c50 2023-09-30T00:20:55.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.251+0000 7fc0bdffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:55.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.252+0000 7fc0bdffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:55.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.252+0000 7fc0bdffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:55.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.252+0000 7fc0bdffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:55.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.252+0000 7fc0bdffb700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc0ac098780 0x7fc0ac09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:55.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.252+0000 7fc0cd071700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc0ac098780 0x7fc0ac09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:55.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.252+0000 7fc0bdffb700 1 -- 172.21.15.94:0/1561197088 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fc0c4021070 con 0x7fc0c8070c50 2023-09-30T00:20:55.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.252+0000 7fc0cd071700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc0ac098780 0x7fc0ac09ac30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc0c813ccd0 tx=0x7fc0b80090f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:55.256 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.255+0000 7fc0bdffb700 1 -- 172.21.15.94:0/1561197088 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc0c408fc10 con 0x7fc0c8070c50 2023-09-30T00:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.479+0000 7fc0cfad6700 1 -- 172.21.15.94:0/1561197088 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc0c80a1310 con 0x7fc0c8070c50 2023-09-30T00:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.480+0000 7fc0bdffb700 1 -- 172.21.15.94:0/1561197088 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc0c80a1310 con 0x7fc0c8070c50 2023-09-30T00:20:55.480 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:55.481 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:55.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 10 mgrc shutdown 2023-09-30T00:20:55.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 1 -- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc0ac098780 msgr2=0x7fc0ac09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:55.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc0ac098780 0x7fc0ac09ac30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc0c813ccd0 tx=0x7fc0b80090f0 comp rx=0 tx=0).stop 2023-09-30T00:20:55.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 1 -- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 msgr2=0x7fc0c8141b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:55.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 0x7fc0c8141b20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009290 tx=0x7fc0c4016000 comp rx=0 tx=0).stop 2023-09-30T00:20:55.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 1 -- 172.21.15.94:0/1561197088 shutdown_connections 2023-09-30T00:20:55.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc0ac098780 0x7fc0ac09ac30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:55.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc0c8070c50 0x7fc0c8141b20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:55.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0c813fd70 0x7fc0c813a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:55.484 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 1 --2- 172.21.15.94:0/1561197088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0c80b7160 0x7fc0c809f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:55.484 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 1 -- 172.21.15.94:0/1561197088 >> 172.21.15.94:0/1561197088 conn(0x7fc0c806a090 msgr2=0x7fc0c80702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:55.484 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 1 -- 172.21.15.94:0/1561197088 shutdown_connections 2023-09-30T00:20:55.484 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:55.482+0000 7fc0cfad6700 1 -- 172.21.15.94:0/1561197088 wait complete. 2023-09-30T00:20:58.494 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:20:58.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.617+0000 7f47f6adf700 1 Processor -- start 2023-09-30T00:20:58.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.617+0000 7f47f6adf700 1 -- start start 2023-09-30T00:20:58.623 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f6adf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 0x7f47f00a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:58.623 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f6adf700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47f013fd70 0x7f47f01401f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:58.623 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f6adf700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f00a1310 con 0x7f47f00b7160 2023-09-30T00:20:58.623 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f6adf700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f0071020 con 0x7f47f013fd70 2023-09-30T00:20:58.623 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f6adf700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00b76a0 con 0x7f47f0070c50 2023-09-30T00:20:58.623 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f487b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 0x7f47f00a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:58.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f507c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47f013fd70 0x7f47f01401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:58.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f507c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47f013fd70 0x7f47f01401f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44930/0 (socket says 172.21.15.94:44930) 2023-09-30T00:20:58.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f507c700 1 -- 172.21.15.94:0/1627123684 learned_addr learned my addr 172.21.15.94:0/1627123684 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:58.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47e7fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f47f00b7160 0x7f47f00a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58870/0 (socket says 172.21.15.94:58870) 2023-09-30T00:20:58.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f487b700 1 -- 172.21.15.94:0/1627123684 >> v2:172.21.15.94:3300/0 conn(0x7f47f013fd70 msgr2=0x7f47f01401f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:20:58.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f487b700 1 --2- 172.21.15.94:0/1627123684 >> v2:172.21.15.94:3300/0 conn(0x7f47f013fd70 0x7f47f01401f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:58.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f487b700 1 -- 172.21.15.94:0/1627123684 >> v1:172.21.15.94:6789/0 conn(0x7f47f00b7160 legacy=0x7f47f00a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:58.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f487b700 1 -- 172.21.15.94:0/1627123684 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f00b7940 con 0x7f47f0070c50 2023-09-30T00:20:58.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.618+0000 7f47f507c700 1 --2- 172.21.15.94:0/1627123684 >> v2:172.21.15.94:3300/0 conn(0x7f47f013fd70 0x7f47f01401f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:20:58.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.619+0000 7f47f487b700 1 --2- 172.21.15.94:0/1627123684 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 0x7f47f00a0dd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f47ec0098e0 tx=0x7f47ec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10e36f14c3962e0c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:58.624 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.619+0000 7f47e77fe700 1 -- 172.21.15.94:0/1627123684 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec00b6d0 con 0x7f47f0070c50 2023-09-30T00:20:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.619+0000 7f47e77fe700 1 -- 172.21.15.94:0/1627123684 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec00b830 con 0x7f47f0070c50 2023-09-30T00:20:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.619+0000 7f47e77fe700 1 -- 172.21.15.94:0/1627123684 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec00bbd0 con 0x7f47f0070c50 2023-09-30T00:20:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.619+0000 7f47f6adf700 1 -- 172.21.15.94:0/1627123684 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 msgr2=0x7f47f00a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.619+0000 7f47f6adf700 1 --2- 172.21.15.94:0/1627123684 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 0x7f47f00a0dd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f47ec0098e0 tx=0x7f47ec0091a0 comp rx=0 tx=0).stop 2023-09-30T00:20:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.619+0000 7f47f6adf700 1 -- 172.21.15.94:0/1627123684 shutdown_connections 2023-09-30T00:20:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.619+0000 7f47f6adf700 1 --2- 172.21.15.94:0/1627123684 >> v2:172.21.15.94:3300/0 conn(0x7f47f013fd70 0x7f47f01401f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.619+0000 7f47f6adf700 1 --2- 172.21.15.94:0/1627123684 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 0x7f47f00a0dd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.619+0000 7f47f6adf700 1 -- 172.21.15.94:0/1627123684 >> 172.21.15.94:0/1627123684 conn(0x7f47f006a090 msgr2=0x7f47f00702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.619+0000 7f47f6adf700 1 -- 172.21.15.94:0/1627123684 shutdown_connections 2023-09-30T00:20:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.619+0000 7f47f6adf700 1 -- 172.21.15.94:0/1627123684 wait complete. 2023-09-30T00:20:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.620+0000 7f47f6adf700 1 Processor -- start 2023-09-30T00:20:58.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.620+0000 7f47f6adf700 1 -- start start 2023-09-30T00:20:58.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f6adf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 0x7f47f0135a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:58.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f6adf700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47f00b7160 0x7f47f00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:58.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f6adf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47f013fd70 0x7f47f013b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:58.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f6adf700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00a4680 con 0x7f47f00b7160 2023-09-30T00:20:58.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f6adf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f01367d0 con 0x7f47f013fd70 2023-09-30T00:20:58.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f6adf700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00a5800 con 0x7f47f0070c50 2023-09-30T00:20:58.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f487b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 0x7f47f0135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:58.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47e7fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47f00b7160 0x7f47f00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:58.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f487b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 0x7f47f0135a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:33290/0 (socket says 172.21.15.94:33290) 2023-09-30T00:20:58.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f487b700 1 -- 172.21.15.94:0/3422767008 learned_addr learned my addr 172.21.15.94:0/3422767008 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:20:58.626 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f487b700 1 -- 172.21.15.94:0/3422767008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47f013fd70 msgr2=0x7f47f013b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:58.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f487b700 1 --2- 172.21.15.94:0/3422767008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47f013fd70 0x7f47f013b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:58.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f487b700 1 -- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47f00b7160 msgr2=0x7f47f00a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:58.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f487b700 1 --2- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47f00b7160 0x7f47f00a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:58.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f487b700 1 -- 172.21.15.94:0/3422767008 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e8009ae0 con 0x7f47f0070c50 2023-09-30T00:20:58.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.621+0000 7f47f487b700 1 --2- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 0x7f47f0135a40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f47f00a9730 tx=0x7f47ec017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:58.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.622+0000 7f47e4ff9700 1 -- 172.21.15.94:0/3422767008 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec01bca0 con 0x7f47f0070c50 2023-09-30T00:20:58.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.622+0000 7f47e4ff9700 1 -- 172.21.15.94:0/3422767008 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec01be00 con 0x7f47f0070c50 2023-09-30T00:20:58.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.622+0000 7f47f6adf700 1 -- 172.21.15.94:0/3422767008 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec009420 con 0x7f47f0070c50 2023-09-30T00:20:58.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.622+0000 7f47e4ff9700 1 -- 172.21.15.94:0/3422767008 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec011670 con 0x7f47f0070c50 2023-09-30T00:20:58.627 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.622+0000 7f47f6adf700 1 -- 172.21.15.94:0/3422767008 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f0136170 con 0x7f47f0070c50 2023-09-30T00:20:58.628 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.623+0000 7f47f6adf700 1 -- 172.21.15.94:0/3422767008 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f0071630 con 0x7f47f0070c50 2023-09-30T00:20:58.628 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.623+0000 7f47e4ff9700 1 -- 172.21.15.94:0/3422767008 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f47ec011a20 con 0x7f47f0070c50 2023-09-30T00:20:58.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.623+0000 7f47e4ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:20:58.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.624+0000 7f47e4ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:20:58.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.624+0000 7f47e4ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:58.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.624+0000 7f47e4ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:20:58.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.624+0000 7f47e4ff9700 1 --2- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f47dc0987b0 0x7f47dc09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:20:58.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.624+0000 7f47e7fff700 1 --2- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f47dc0987b0 0x7f47dc09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:20:58.629 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.624+0000 7f47e4ff9700 1 -- 172.21.15.94:0/3422767008 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f47ec0c1d90 con 0x7f47f0070c50 2023-09-30T00:20:58.630 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.624+0000 7f47e7fff700 1 --2- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f47dc0987b0 0x7f47dc09ac60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f47f0140360 tx=0x7f47e0007eb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:20:58.630 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.627+0000 7f47e4ff9700 1 -- 172.21.15.94:0/3422767008 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f47ec08f460 con 0x7f47f0070c50 2023-09-30T00:20:58.851 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.850+0000 7f47f6adf700 1 -- 172.21.15.94:0/3422767008 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f47f00a2070 con 0x7f47f0070c50 2023-09-30T00:20:58.851 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.851+0000 7f47e4ff9700 1 -- 172.21.15.94:0/3422767008 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f47ec092a80 con 0x7f47f0070c50 2023-09-30T00:20:58.851 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:20:58.852 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:20:58.853 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.852+0000 7f47f6adf700 10 mgrc shutdown 2023-09-30T00:20:58.853 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.852+0000 7f47f6adf700 1 -- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f47dc0987b0 msgr2=0x7f47dc09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:58.854 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.852+0000 7f47f6adf700 1 --2- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f47dc0987b0 0x7f47dc09ac60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f47f0140360 tx=0x7f47e0007eb0 comp rx=0 tx=0).stop 2023-09-30T00:20:58.854 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.852+0000 7f47f6adf700 1 -- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 msgr2=0x7f47f0135a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:20:58.854 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.852+0000 7f47f6adf700 1 --2- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 0x7f47f0135a40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f47f00a9730 tx=0x7f47ec017000 comp rx=0 tx=0).stop 2023-09-30T00:20:58.854 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.852+0000 7f47f6adf700 1 -- 172.21.15.94:0/3422767008 shutdown_connections 2023-09-30T00:20:58.854 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.852+0000 7f47f6adf700 1 --2- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f47dc0987b0 0x7f47dc09ac60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:58.854 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.852+0000 7f47f6adf700 1 --2- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47f0070c50 0x7f47f0135a40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:58.854 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.852+0000 7f47f6adf700 1 --2- 172.21.15.94:0/3422767008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47f013fd70 0x7f47f013b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:58.854 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.852+0000 7f47f6adf700 1 --2- 172.21.15.94:0/3422767008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47f00b7160 0x7f47f00a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:20:58.854 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.852+0000 7f47f6adf700 1 -- 172.21.15.94:0/3422767008 >> 172.21.15.94:0/3422767008 conn(0x7f47f006a090 msgr2=0x7f47f013d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:20:58.855 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.853+0000 7f47f6adf700 1 -- 172.21.15.94:0/3422767008 shutdown_connections 2023-09-30T00:20:58.855 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:20:58.853+0000 7f47f6adf700 1 -- 172.21.15.94:0/3422767008 wait complete. 2023-09-30T00:21:01.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:01.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.986+0000 7f067ff83700 1 Processor -- start 2023-09-30T00:21:01.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.986+0000 7f067ff83700 1 -- start start 2023-09-30T00:21:01.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067ff83700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0678070c50 0x7f06780a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:01.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067ff83700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f06780b7160 0x7f06780a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:01.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067ff83700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06780a1310 con 0x7f067813fd70 2023-09-30T00:21:01.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067ff83700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0678071020 con 0x7f06780b7160 2023-09-30T00:21:01.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067ff83700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06780b76a0 con 0x7f0678070c50 2023-09-30T00:21:01.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067d51e700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f06780b7160 0x7f06780a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:01.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067e520700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f067813fd70 0x7f06780b4dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58878/0 (socket says 172.21.15.94:58878) 2023-09-30T00:21:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067e520700 1 -- 172.21.15.94:0/3904149468 learned_addr learned my addr 172.21.15.94:0/3904149468 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:21:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067dd1f700 1 --2- 172.21.15.94:0/3904149468 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0678070c50 0x7f06780a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067d51e700 1 -- 172.21.15.94:0/3904149468 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0678070c50 msgr2=0x7f06780a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067d51e700 1 --2- 172.21.15.94:0/3904149468 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0678070c50 0x7f06780a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067d51e700 1 -- 172.21.15.94:0/3904149468 >> v1:172.21.15.94:6789/0 conn(0x7f067813fd70 legacy=0x7f06780b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067d51e700 1 -- 172.21.15.94:0/3904149468 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06780b7940 con 0x7f06780b7160 2023-09-30T00:21:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.987+0000 7f067cd1d700 1 -- 172.21.15.94:0/3904149468 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074640852 0 0) 0x7f0670001490 con 0x7f067813fd70 2023-09-30T00:21:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067dd1f700 1 --2- 172.21.15.94:0/3904149468 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0678070c50 0x7f06780a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067d51e700 1 --2- 172.21.15.94:0/3904149468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06780b7160 0x7f06780a2530 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f06680098e0 tx=0x7f06680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336cc95f887ecc27 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067cd1d700 1 -- 172.21.15.94:0/3904149468 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066800b6d0 con 0x7f06780b7160 2023-09-30T00:21:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067cd1d700 1 -- 172.21.15.94:0/3904149468 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066800b830 con 0x7f06780b7160 2023-09-30T00:21:01.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067cd1d700 1 -- 172.21.15.94:0/3904149468 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066800bbd0 con 0x7f06780b7160 2023-09-30T00:21:01.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067ff83700 1 -- 172.21.15.94:0/3904149468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06780b7160 msgr2=0x7f06780a2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:01.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067ff83700 1 --2- 172.21.15.94:0/3904149468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06780b7160 0x7f06780a2530 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f06680098e0 tx=0x7f06680091a0 comp rx=0 tx=0).stop 2023-09-30T00:21:01.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067ff83700 1 -- 172.21.15.94:0/3904149468 shutdown_connections 2023-09-30T00:21:01.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067ff83700 1 --2- 172.21.15.94:0/3904149468 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0678070c50 0x7f06780a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:01.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067ff83700 1 --2- 172.21.15.94:0/3904149468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06780b7160 0x7f06780a2530 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:01.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067ff83700 1 -- 172.21.15.94:0/3904149468 >> 172.21.15.94:0/3904149468 conn(0x7f067806a090 msgr2=0x7f06780702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:01.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067ff83700 1 -- 172.21.15.94:0/3904149468 shutdown_connections 2023-09-30T00:21:01.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.988+0000 7f067ff83700 1 -- 172.21.15.94:0/3904149468 wait complete. 2023-09-30T00:21:01.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.989+0000 7f067ff83700 1 Processor -- start 2023-09-30T00:21:01.992 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.989+0000 7f067ff83700 1 -- start start 2023-09-30T00:21:01.995 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067ff83700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0678070c50 0x7f06780ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:01.995 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067ff83700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06780b7160 0x7f0678150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:01.995 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067ff83700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f067813fd70 0x7f06781510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:01.995 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067ff83700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0678151620 con 0x7f06780b7160 2023-09-30T00:21:01.995 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067ff83700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0678150780 con 0x7f0678070c50 2023-09-30T00:21:01.995 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067ff83700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06780b3be0 con 0x7f067813fd70 2023-09-30T00:21:01.996 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067d51e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06780b7160 0x7f0678150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:01.996 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067e520700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f067813fd70 0x7f06781510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:01.996 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067d51e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06780b7160 0x7f0678150ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44966/0 (socket says 172.21.15.94:44966) 2023-09-30T00:21:01.996 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067d51e700 1 -- 172.21.15.94:0/1285148061 learned_addr learned my addr 172.21.15.94:0/1285148061 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:01.996 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067dd1f700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0678070c50 0x7f06780ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:01.996 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067d51e700 1 -- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f067813fd70 msgr2=0x7f06781510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:01.996 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067d51e700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f067813fd70 0x7f06781510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:01.997 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067d51e700 1 -- 172.21.15.94:0/1285148061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0678070c50 msgr2=0x7f06780ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:01.997 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067d51e700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0678070c50 0x7f06780ac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:01.997 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067d51e700 1 -- 172.21.15.94:0/1285148061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0674009420 con 0x7f06780b7160 2023-09-30T00:21:01.997 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067dd1f700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0678070c50 0x7f06780ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:21:01.997 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067e520700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f067813fd70 0x7f06781510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:01.997 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067d51e700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06780b7160 0x7f0678150ba0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0668009f80 tx=0x7f0668017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:01.998 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f066dffb700 1 -- 172.21.15.94:0/1285148061 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066800bbb0 con 0x7f06780b7160 2023-09-30T00:21:01.998 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f066dffb700 1 -- 172.21.15.94:0/1285148061 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0668011420 con 0x7f06780b7160 2023-09-30T00:21:01.998 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.990+0000 7f067ff83700 1 -- 172.21.15.94:0/1285148061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0668009420 con 0x7f06780b7160 2023-09-30T00:21:01.998 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.991+0000 7f066dffb700 1 -- 172.21.15.94:0/1285148061 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06680117c0 con 0x7f06780b7160 2023-09-30T00:21:01.998 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.991+0000 7f067ff83700 1 -- 172.21.15.94:0/1285148061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06780a8190 con 0x7f06780b7160 2023-09-30T00:21:01.998 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.992+0000 7f067ff83700 1 -- 172.21.15.94:0/1285148061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06780ace50 con 0x7f06780b7160 2023-09-30T00:21:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.992+0000 7f066dffb700 1 -- 172.21.15.94:0/1285148061 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f0668011b30 con 0x7f06780b7160 2023-09-30T00:21:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.992+0000 7f066dffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.993+0000 7f066dffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.993+0000 7f066dffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.993+0000 7f066dffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.993+0000 7f066dffb700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f065c098780 0x7f065c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.993+0000 7f067dd1f700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f065c098780 0x7f065c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.993+0000 7f066dffb700 1 -- 172.21.15.94:0/1285148061 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f06680c1b50 con 0x7f06780b7160 2023-09-30T00:21:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.993+0000 7f067dd1f700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f065c098780 0x7f065c09ac30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f06740093f0 tx=0x7f0674008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:01.997+0000 7f066dffb700 1 -- 172.21.15.94:0/1285148061 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f066808f220 con 0x7f06780b7160 2023-09-30T00:21:02.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.216+0000 7f067ff83700 1 -- 172.21.15.94:0/1285148061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f06780ad030 con 0x7f06780b7160 2023-09-30T00:21:02.218 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.217+0000 7f066dffb700 1 -- 172.21.15.94:0/1285148061 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f06780ad030 con 0x7f06780b7160 2023-09-30T00:21:02.218 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:02.218 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:02.220 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 10 mgrc shutdown 2023-09-30T00:21:02.220 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 1 -- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f065c098780 msgr2=0x7f065c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:02.220 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f065c098780 0x7f065c09ac30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f06740093f0 tx=0x7f0674008040 comp rx=0 tx=0).stop 2023-09-30T00:21:02.220 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 1 -- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06780b7160 msgr2=0x7f0678150ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:02.220 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06780b7160 0x7f0678150ba0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0668009f80 tx=0x7f0668017000 comp rx=0 tx=0).stop 2023-09-30T00:21:02.220 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 1 -- 172.21.15.94:0/1285148061 shutdown_connections 2023-09-30T00:21:02.221 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f065c098780 0x7f065c09ac30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:02.221 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f067813fd70 0x7f06781510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:02.221 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0678070c50 0x7f06780ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:02.221 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 1 --2- 172.21.15.94:0/1285148061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06780b7160 0x7f0678150ba0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:02.221 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 1 -- 172.21.15.94:0/1285148061 >> 172.21.15.94:0/1285148061 conn(0x7f067806a090 msgr2=0x7f06780b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:02.221 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 1 -- 172.21.15.94:0/1285148061 shutdown_connections 2023-09-30T00:21:02.221 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:02.219+0000 7f067ff83700 1 -- 172.21.15.94:0/1285148061 wait complete. 2023-09-30T00:21:05.232 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:05.359 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.359+0000 7ff09e980700 1 Processor -- start 2023-09-30T00:21:05.359 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.359+0000 7ff09e980700 1 -- start start 2023-09-30T00:21:05.361 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.359+0000 7ff09e980700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980a4840 0x7ff0980a5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:05.361 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.359+0000 7ff09e980700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0980a6fc0 0x7ff098148e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:05.361 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.359+0000 7ff09e980700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0980a6050 con 0x7ff0980a4470 2023-09-30T00:21:05.361 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.359+0000 7ff09e980700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0980b4dd0 con 0x7ff0980a6fc0 2023-09-30T00:21:05.361 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.359+0000 7ff09e980700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09806ea50 con 0x7ff0980a4840 2023-09-30T00:21:05.361 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.359+0000 7ff0977fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980a4840 0x7ff0980a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:05.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.359+0000 7ff097fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff0980a4470 0x7ff0980b5550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41118/0 (socket says 172.21.15.94:41118) 2023-09-30T00:21:05.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.359+0000 7ff097fff700 1 -- 172.21.15.94:0/3027174822 learned_addr learned my addr 172.21.15.94:0/3027174822 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:21:05.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.360+0000 7ff09cf1d700 1 --2- 172.21.15.94:0/3027174822 >> v2:172.21.15.121:3300/0 conn(0x7ff0980a6fc0 0x7ff098148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:05.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.360+0000 7ff096ffd700 1 -- 172.21.15.94:0/3027174822 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079507969 0 0) 0x7ff0980a6050 con 0x7ff0980a4470 2023-09-30T00:21:05.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.360+0000 7ff0977fe700 1 -- 172.21.15.94:0/3027174822 >> v2:172.21.15.121:3300/0 conn(0x7ff0980a6fc0 msgr2=0x7ff098148e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:05.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.360+0000 7ff0977fe700 1 --2- 172.21.15.94:0/3027174822 >> v2:172.21.15.121:3300/0 conn(0x7ff0980a6fc0 0x7ff098148e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:05.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.360+0000 7ff0977fe700 1 -- 172.21.15.94:0/3027174822 >> v1:172.21.15.94:6789/0 conn(0x7ff0980a4470 legacy=0x7ff0980b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:05.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.360+0000 7ff0977fe700 1 -- 172.21.15.94:0/3027174822 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0980a5c60 con 0x7ff0980a4840 2023-09-30T00:21:05.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.360+0000 7ff0977fe700 1 --2- 172.21.15.94:0/3027174822 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980a4840 0x7ff0980a5720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff08c0098e0 tx=0x7ff08c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d2db4c9d2bb0691 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:05.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.360+0000 7ff096ffd700 1 -- 172.21.15.94:0/3027174822 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c009e80 con 0x7ff0980a4840 2023-09-30T00:21:05.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.360+0000 7ff096ffd700 1 -- 172.21.15.94:0/3027174822 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08c00b860 con 0x7ff0980a4840 2023-09-30T00:21:05.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.360+0000 7ff096ffd700 1 -- 172.21.15.94:0/3027174822 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c00bbe0 con 0x7ff0980a4840 2023-09-30T00:21:05.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.360+0000 7ff09e980700 1 -- 172.21.15.94:0/3027174822 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980a4840 msgr2=0x7ff0980a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:05.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.360+0000 7ff09e980700 1 --2- 172.21.15.94:0/3027174822 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980a4840 0x7ff0980a5720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff08c0098e0 tx=0x7ff08c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:21:05.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.361+0000 7ff09e980700 1 -- 172.21.15.94:0/3027174822 shutdown_connections 2023-09-30T00:21:05.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.361+0000 7ff09e980700 1 --2- 172.21.15.94:0/3027174822 >> v2:172.21.15.121:3300/0 conn(0x7ff0980a6fc0 0x7ff098148e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:05.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.361+0000 7ff09e980700 1 --2- 172.21.15.94:0/3027174822 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980a4840 0x7ff0980a5720 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:05.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.361+0000 7ff09e980700 1 -- 172.21.15.94:0/3027174822 >> 172.21.15.94:0/3027174822 conn(0x7ff09806cbf0 msgr2=0x7ff09806cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:05.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.361+0000 7ff09e980700 1 -- 172.21.15.94:0/3027174822 shutdown_connections 2023-09-30T00:21:05.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.361+0000 7ff09e980700 1 -- 172.21.15.94:0/3027174822 wait complete. 2023-09-30T00:21:05.365 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.361+0000 7ff09e980700 1 Processor -- start 2023-09-30T00:21:05.365 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.362+0000 7ff09e980700 1 -- start start 2023-09-30T00:21:05.365 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.362+0000 7ff09e980700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0980a4470 0x7ff098145730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:05.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.362+0000 7ff09e980700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0980a4840 0x7ff098068c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:05.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.362+0000 7ff09e980700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980a6fc0 0x7ff0981380a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:05.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.362+0000 7ff09e980700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09813fdd0 con 0x7ff0980a4470 2023-09-30T00:21:05.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.362+0000 7ff09e980700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff098069f80 con 0x7ff0980a4840 2023-09-30T00:21:05.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.362+0000 7ff09e980700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0980697a0 con 0x7ff0980a6fc0 2023-09-30T00:21:05.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.362+0000 7ff097fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0980a4470 0x7ff098145730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:05.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.362+0000 7ff097fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0980a4470 0x7ff098145730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53614/0 (socket says 172.21.15.94:53614) 2023-09-30T00:21:05.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.362+0000 7ff097fff700 1 -- 172.21.15.94:0/2591565474 learned_addr learned my addr 172.21.15.94:0/2591565474 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:05.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.362+0000 7ff09cf1d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980a6fc0 0x7ff0981380a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:05.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.362+0000 7ff0977fe700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0980a4840 0x7ff098068c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:05.367 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.363+0000 7ff09cf1d700 1 -- 172.21.15.94:0/2591565474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0980a4840 msgr2=0x7ff098068c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:05.367 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.363+0000 7ff09cf1d700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0980a4840 0x7ff098068c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:05.367 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.363+0000 7ff09cf1d700 1 -- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0980a4470 msgr2=0x7ff098145730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:05.367 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.363+0000 7ff09cf1d700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0980a4470 0x7ff098145730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:05.367 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.363+0000 7ff09cf1d700 1 -- 172.21.15.94:0/2591565474 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c009420 con 0x7ff0980a6fc0 2023-09-30T00:21:05.367 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.363+0000 7ff097fff700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0980a4470 0x7ff098145730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:05.367 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.363+0000 7ff09cf1d700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980a6fc0 0x7ff0981380a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff090007ed0 tx=0x7ff090009850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:05.367 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.363+0000 7ff07ffff700 1 -- 172.21.15.94:0/2591565474 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff090018070 con 0x7ff0980a6fc0 2023-09-30T00:21:05.367 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.363+0000 7ff07ffff700 1 -- 172.21.15.94:0/2591565474 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09000eb90 con 0x7ff0980a6fc0 2023-09-30T00:21:05.367 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.363+0000 7ff09e980700 1 -- 172.21.15.94:0/2591565474 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff09806a230 con 0x7ff0980a6fc0 2023-09-30T00:21:05.367 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.363+0000 7ff07ffff700 1 -- 172.21.15.94:0/2591565474 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff090014420 con 0x7ff0980a6fc0 2023-09-30T00:21:05.367 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.363+0000 7ff09e980700 1 -- 172.21.15.94:0/2591565474 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09813f890 con 0x7ff0980a6fc0 2023-09-30T00:21:05.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.364+0000 7ff09e980700 1 -- 172.21.15.94:0/2591565474 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0981367d0 con 0x7ff0980a6fc0 2023-09-30T00:21:05.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.365+0000 7ff07ffff700 1 -- 172.21.15.94:0/2591565474 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ff090003020 con 0x7ff0980a6fc0 2023-09-30T00:21:05.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.365+0000 7ff07ffff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:05.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.365+0000 7ff07ffff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:05.368 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.365+0000 7ff07ffff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:05.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.365+0000 7ff07ffff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:05.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.365+0000 7ff07ffff700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff084098800 0x7ff08409acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:05.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.366+0000 7ff097fff700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff084098800 0x7ff08409acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:05.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.366+0000 7ff07ffff700 1 -- 172.21.15.94:0/2591565474 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7ff0900c1520 con 0x7ff0980a6fc0 2023-09-30T00:21:05.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.366+0000 7ff097fff700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff084098800 0x7ff08409acb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff0880017b0 tx=0x7ff088001070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:05.369 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.369+0000 7ff07ffff700 1 -- 172.21.15.94:0/2591565474 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff09008eb50 con 0x7ff0980a6fc0 2023-09-30T00:21:05.591 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.590+0000 7ff09e980700 1 -- 172.21.15.94:0/2591565474 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff09806fbd0 con 0x7ff0980a6fc0 2023-09-30T00:21:05.591 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.591+0000 7ff07ffff700 1 -- 172.21.15.94:0/2591565474 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff090092170 con 0x7ff0980a6fc0 2023-09-30T00:21:05.591 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:05.592 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:05.593 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 10 mgrc shutdown 2023-09-30T00:21:05.593 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 1 -- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff084098800 msgr2=0x7ff08409acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:05.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff084098800 0x7ff08409acb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff0880017b0 tx=0x7ff088001070 comp rx=0 tx=0).stop 2023-09-30T00:21:05.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 1 -- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980a6fc0 msgr2=0x7ff0981380a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:05.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980a6fc0 0x7ff0981380a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff090007ed0 tx=0x7ff090009850 comp rx=0 tx=0).stop 2023-09-30T00:21:05.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 1 -- 172.21.15.94:0/2591565474 shutdown_connections 2023-09-30T00:21:05.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff084098800 0x7ff08409acb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:05.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980a6fc0 0x7ff0981380a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:05.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0980a4470 0x7ff098145730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:05.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 1 --2- 172.21.15.94:0/2591565474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0980a4840 0x7ff098068c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:05.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 1 -- 172.21.15.94:0/2591565474 >> 172.21.15.94:0/2591565474 conn(0x7ff09806cbf0 msgr2=0x7ff09806cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:05.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 1 -- 172.21.15.94:0/2591565474 shutdown_connections 2023-09-30T00:21:05.594 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:05.593+0000 7ff09e980700 1 -- 172.21.15.94:0/2591565474 wait complete. 2023-09-30T00:21:08.606 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:08.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.731+0000 7f65b6ef2700 1 Processor -- start 2023-09-30T00:21:08.732 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.731+0000 7f65b6ef2700 1 -- start start 2023-09-30T00:21:08.734 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.732+0000 7f65b6ef2700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f65b00a4470 0x7f65b00b5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:08.734 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.732+0000 7f65b6ef2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 0x7f65b00a5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:08.734 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.732+0000 7f65b6ef2700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b00a6050 con 0x7f65b00a6fc0 2023-09-30T00:21:08.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.732+0000 7f65b6ef2700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b00b4dd0 con 0x7f65b00a4470 2023-09-30T00:21:08.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.732+0000 7f65b6ef2700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b006ea50 con 0x7f65b00a4840 2023-09-30T00:21:08.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.732+0000 7f65a7fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 0x7f65b00a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:08.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.732+0000 7f65b548f700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f65b00a6fc0 0x7f65b0141a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41120/0 (socket says 172.21.15.94:41120) 2023-09-30T00:21:08.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.732+0000 7f65b548f700 1 -- 172.21.15.94:0/3087994006 learned_addr learned my addr 172.21.15.94:0/3087994006 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:21:08.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.732+0000 7f65b4c8e700 1 --2- 172.21.15.94:0/3087994006 >> v2:172.21.15.121:3300/0 conn(0x7f65b00a4470 0x7f65b00b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:08.735 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.732+0000 7f65a7fff700 1 -- 172.21.15.94:0/3087994006 >> v2:172.21.15.121:3300/0 conn(0x7f65b00a4470 msgr2=0x7f65b00b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:08.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65a7fff700 1 --2- 172.21.15.94:0/3087994006 >> v2:172.21.15.121:3300/0 conn(0x7f65b00a4470 0x7f65b00b5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:08.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.732+0000 7f65a77fe700 1 -- 172.21.15.94:0/3087994006 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301186211 0 0) 0x7f65b00a6050 con 0x7f65b00a6fc0 2023-09-30T00:21:08.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65a7fff700 1 -- 172.21.15.94:0/3087994006 >> v1:172.21.15.94:6789/0 conn(0x7f65b00a6fc0 legacy=0x7f65b0141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:08.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65a7fff700 1 -- 172.21.15.94:0/3087994006 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b00b5a90 con 0x7f65b00a4840 2023-09-30T00:21:08.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65a7fff700 1 --2- 172.21.15.94:0/3087994006 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 0x7f65b00a5720 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f65a00098e0 tx=0x7f65a00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23315ed36fcc0963 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:08.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65a77fe700 1 -- 172.21.15.94:0/3087994006 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a000b640 con 0x7f65b00a4840 2023-09-30T00:21:08.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65a77fe700 1 -- 172.21.15.94:0/3087994006 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a000b7a0 con 0x7f65b00a4840 2023-09-30T00:21:08.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65a77fe700 1 -- 172.21.15.94:0/3087994006 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a000bb40 con 0x7f65b00a4840 2023-09-30T00:21:08.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3087994006 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 msgr2=0x7f65b00a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:08.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65b6ef2700 1 --2- 172.21.15.94:0/3087994006 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 0x7f65b00a5720 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f65a00098e0 tx=0x7f65a00091a0 comp rx=0 tx=0).stop 2023-09-30T00:21:08.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3087994006 shutdown_connections 2023-09-30T00:21:08.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65b6ef2700 1 --2- 172.21.15.94:0/3087994006 >> v2:172.21.15.121:3300/0 conn(0x7f65b00a4470 0x7f65b00b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:08.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65b6ef2700 1 --2- 172.21.15.94:0/3087994006 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 0x7f65b00a5720 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:08.738 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3087994006 >> 172.21.15.94:0/3087994006 conn(0x7f65b006cbf0 msgr2=0x7f65b006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:08.738 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3087994006 shutdown_connections 2023-09-30T00:21:08.738 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.733+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3087994006 wait complete. 2023-09-30T00:21:08.741 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.734+0000 7f65b6ef2700 1 Processor -- start 2023-09-30T00:21:08.741 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65b6ef2700 1 -- start start 2023-09-30T00:21:08.741 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65b6ef2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65b00a4470 0x7f65b0069770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:08.741 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65b6ef2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 0x7f65b00ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:08.741 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65b6ef2700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65b00a6fc0 0x7f65b0145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:08.741 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65b6ef2700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b0069cb0 con 0x7f65b00a6fc0 2023-09-30T00:21:08.741 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65b6ef2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b0069ea0 con 0x7f65b00a4470 2023-09-30T00:21:08.742 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65b6ef2700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b006a160 con 0x7f65b00a4840 2023-09-30T00:21:08.742 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65a7fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 0x7f65b00ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:08.742 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65b548f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65b00a6fc0 0x7f65b0145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:08.742 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65a7fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 0x7f65b00ae190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:59632/0 (socket says 172.21.15.94:59632) 2023-09-30T00:21:08.742 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65b4c8e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65b00a4470 0x7f65b0069770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:08.742 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65a7fff700 1 -- 172.21.15.94:0/3233855987 learned_addr learned my addr 172.21.15.94:0/3233855987 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:08.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65a7fff700 1 -- 172.21.15.94:0/3233855987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65b00a4470 msgr2=0x7f65b0069770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:08.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65a7fff700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65b00a4470 0x7f65b0069770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:08.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.735+0000 7f65a7fff700 1 -- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65b00a6fc0 msgr2=0x7f65b0145e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:08.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.736+0000 7f65a7fff700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65b00a6fc0 0x7f65b0145e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:08.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.736+0000 7f65a7fff700 1 -- 172.21.15.94:0/3233855987 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a0009420 con 0x7f65b00a4840 2023-09-30T00:21:08.743 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.736+0000 7f65b548f700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65b00a6fc0 0x7f65b0145e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:08.744 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.736+0000 7f65a7fff700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 0x7f65b00ae190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f65b006eb90 tx=0x7f65a0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:08.744 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.736+0000 7f65a4ff9700 1 -- 172.21.15.94:0/3233855987 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a000bb20 con 0x7f65b00a4840 2023-09-30T00:21:08.744 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.736+0000 7f65a4ff9700 1 -- 172.21.15.94:0/3233855987 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a0000c90 con 0x7f65b00a4840 2023-09-30T00:21:08.744 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.736+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3233855987 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b006a420 con 0x7f65b00a4840 2023-09-30T00:21:08.744 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.736+0000 7f65a4ff9700 1 -- 172.21.15.94:0/3233855987 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a0011630 con 0x7f65b00a4840 2023-09-30T00:21:08.744 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.736+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3233855987 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b00ad360 con 0x7f65b00a4840 2023-09-30T00:21:08.744 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.737+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3233855987 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b006fbd0 con 0x7f65b00a4840 2023-09-30T00:21:08.745 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.738+0000 7f65a4ff9700 1 -- 172.21.15.94:0/3233855987 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f65a00119a0 con 0x7f65b00a4840 2023-09-30T00:21:08.745 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.738+0000 7f65a4ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:08.745 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.738+0000 7f65a4ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:08.745 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.738+0000 7f65a4ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:08.745 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.738+0000 7f65a4ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:08.745 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.738+0000 7f65a4ff9700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f659c098780 0x7f659c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:08.745 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.738+0000 7f65b4c8e700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f659c098780 0x7f659c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:08.746 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.738+0000 7f65a4ff9700 1 -- 172.21.15.94:0/3233855987 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f65a00c17b0 con 0x7f65b00a4840 2023-09-30T00:21:08.746 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.739+0000 7f65b4c8e700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f659c098780 0x7f659c09ac30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f65ac009d10 tx=0x7f65ac0091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:08.746 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.742+0000 7f65a4ff9700 1 -- 172.21.15.94:0/3233855987 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f65a005b5a0 con 0x7f65b00a4840 2023-09-30T00:21:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.961+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3233855987 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f65b00b3c30 con 0x7f65b00a4840 2023-09-30T00:21:09.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.962+0000 7f65a4ff9700 1 -- 172.21.15.94:0/3233855987 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f65a008ee80 con 0x7f65b00a4840 2023-09-30T00:21:09.077 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:09.077 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.963+0000 7f65b6ef2700 10 mgrc shutdown 2023-09-30T00:21:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.963+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f659c098780 msgr2=0x7f659c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.963+0000 7f65b6ef2700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f659c098780 0x7f659c09ac30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f65ac009d10 tx=0x7f65ac0091a0 comp rx=0 tx=0).stop 2023-09-30T00:21:09.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.963+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 msgr2=0x7f65b00ae190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:09.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.963+0000 7f65b6ef2700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 0x7f65b00ae190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f65b006eb90 tx=0x7f65a0017000 comp rx=0 tx=0).stop 2023-09-30T00:21:09.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.963+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3233855987 shutdown_connections 2023-09-30T00:21:09.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.964+0000 7f65b6ef2700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f659c098780 0x7f659c09ac30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:09.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.964+0000 7f65b6ef2700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65b00a6fc0 0x7f65b0145e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:09.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.964+0000 7f65b6ef2700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65b00a4470 0x7f65b0069770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:09.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.964+0000 7f65b6ef2700 1 --2- 172.21.15.94:0/3233855987 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65b00a4840 0x7f65b00ae190 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:09.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.964+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3233855987 >> 172.21.15.94:0/3233855987 conn(0x7f65b006cbf0 msgr2=0x7f65b006cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:09.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.964+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3233855987 shutdown_connections 2023-09-30T00:21:09.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:08.964+0000 7f65b6ef2700 1 -- 172.21.15.94:0/3233855987 wait complete. 2023-09-30T00:21:09.412 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:21:09.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:21:12.084 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:12.208 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.208+0000 7f24adf91700 1 Processor -- start 2023-09-30T00:21:12.208 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.208+0000 7f24adf91700 1 -- start start 2023-09-30T00:21:12.211 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24adf91700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24a81407c0 0x7f24a80b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:12.211 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24adf91700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24a809f880 0x7f24a806e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:12.211 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24adf91700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a8140c60 con 0x7f24a809dbe0 2023-09-30T00:21:12.211 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24adf91700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a8140f20 con 0x7f24a81407c0 2023-09-30T00:21:12.211 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24adf91700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a81411e0 con 0x7f24a809f880 2023-09-30T00:21:12.211 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24a7fff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24a81407c0 0x7f24a80b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:12.212 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24ad790700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24a809f880 0x7f24a806e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:12.212 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24a7fff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24a81407c0 0x7f24a80b1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53632/0 (socket says 172.21.15.94:53632) 2023-09-30T00:21:12.212 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24a7fff700 1 -- 172.21.15.94:0/610954462 learned_addr learned my addr 172.21.15.94:0/610954462 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:12.212 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24ad790700 1 -- 172.21.15.94:0/610954462 >> v2:172.21.15.94:3300/0 conn(0x7f24a81407c0 msgr2=0x7f24a80b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:12.212 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24ad790700 1 --2- 172.21.15.94:0/610954462 >> v2:172.21.15.94:3300/0 conn(0x7f24a81407c0 0x7f24a80b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:12.212 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24ad790700 1 -- 172.21.15.94:0/610954462 >> v1:172.21.15.121:6789/0 conn(0x7f24a809dbe0 legacy=0x7f24a809ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:12.212 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24ad790700 1 -- 172.21.15.94:0/610954462 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a809dfb0 con 0x7f24a809f880 2023-09-30T00:21:12.212 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24a7fff700 1 --2- 172.21.15.94:0/610954462 >> v2:172.21.15.94:3300/0 conn(0x7f24a81407c0 0x7f24a80b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:12.212 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24ad790700 1 --2- 172.21.15.94:0/610954462 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24a809f880 0x7f24a806e8b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f24900098e0 tx=0x7f24900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6076d2f4b085aac server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:12.212 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24a77fe700 1 -- 172.21.15.94:0/610954462 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249000b640 con 0x7f24a809f880 2023-09-30T00:21:12.213 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24a77fe700 1 -- 172.21.15.94:0/610954462 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249000b7a0 con 0x7f24a809f880 2023-09-30T00:21:12.213 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.209+0000 7f24a77fe700 1 -- 172.21.15.94:0/610954462 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249000bb40 con 0x7f24a809f880 2023-09-30T00:21:12.213 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.210+0000 7f24adf91700 1 -- 172.21.15.94:0/610954462 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24a809f880 msgr2=0x7f24a806e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:12.213 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.210+0000 7f24adf91700 1 --2- 172.21.15.94:0/610954462 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24a809f880 0x7f24a806e8b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f24900098e0 tx=0x7f24900091a0 comp rx=0 tx=0).stop 2023-09-30T00:21:12.213 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.210+0000 7f24adf91700 1 -- 172.21.15.94:0/610954462 shutdown_connections 2023-09-30T00:21:12.213 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.210+0000 7f24adf91700 1 --2- 172.21.15.94:0/610954462 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24a809f880 0x7f24a806e8b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:12.213 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.210+0000 7f24adf91700 1 --2- 172.21.15.94:0/610954462 >> v2:172.21.15.94:3300/0 conn(0x7f24a81407c0 0x7f24a80b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:12.213 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.210+0000 7f24adf91700 1 -- 172.21.15.94:0/610954462 >> 172.21.15.94:0/610954462 conn(0x7f24a806ad20 msgr2=0x7f24a80b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:12.213 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.210+0000 7f24adf91700 1 -- 172.21.15.94:0/610954462 shutdown_connections 2023-09-30T00:21:12.213 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.210+0000 7f24adf91700 1 -- 172.21.15.94:0/610954462 wait complete. 2023-09-30T00:21:12.213 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.211+0000 7f24adf91700 1 Processor -- start 2023-09-30T00:21:12.216 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.211+0000 7f24adf91700 1 -- start start 2023-09-30T00:21:12.216 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24adf91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a809dbe0 0x7f24a813c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:12.216 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24adf91700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24a809f880 0x7f24a806d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:12.216 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24adf91700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24a81407c0 0x7f24a806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:12.216 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24adf91700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a813b470 con 0x7f24a809f880 2023-09-30T00:21:12.216 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24adf91700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a806dbf0 con 0x7f24a809dbe0 2023-09-30T00:21:12.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24adf91700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a806cce0 con 0x7f24a81407c0 2023-09-30T00:21:12.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24a7fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24a809f880 0x7f24a806d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:12.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24ad790700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24a81407c0 0x7f24a806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:12.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24a7fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24a809f880 0x7f24a806d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53644/0 (socket says 172.21.15.94:53644) 2023-09-30T00:21:12.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24a7fff700 1 -- 172.21.15.94:0/656174982 learned_addr learned my addr 172.21.15.94:0/656174982 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:12.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24acf8f700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a809dbe0 0x7f24a813c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:12.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24ad790700 1 -- 172.21.15.94:0/656174982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a809dbe0 msgr2=0x7f24a813c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:12.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24ad790700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a809dbe0 0x7f24a813c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:12.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24ad790700 1 -- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24a809f880 msgr2=0x7f24a806d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:12.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24ad790700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24a809f880 0x7f24a806d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:12.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24ad790700 1 -- 172.21.15.94:0/656174982 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24980094f0 con 0x7f24a81407c0 2023-09-30T00:21:12.217 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24a7fff700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24a809f880 0x7f24a806d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:12.218 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24ad790700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24a81407c0 0x7f24a806d6b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2490016000 tx=0x7f2490017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:12.218 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.212+0000 7f24a4ff9700 1 -- 172.21.15.94:0/656174982 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490011420 con 0x7f24a81407c0 2023-09-30T00:21:12.218 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.213+0000 7f24a4ff9700 1 -- 172.21.15.94:0/656174982 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2490011580 con 0x7f24a81407c0 2023-09-30T00:21:12.218 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.213+0000 7f24a4ff9700 1 -- 172.21.15.94:0/656174982 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490011940 con 0x7f24a81407c0 2023-09-30T00:21:12.218 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.213+0000 7f24adf91700 1 -- 172.21.15.94:0/656174982 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2490009420 con 0x7f24a81407c0 2023-09-30T00:21:12.218 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.213+0000 7f24adf91700 1 -- 172.21.15.94:0/656174982 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a806de00 con 0x7f24a81407c0 2023-09-30T00:21:12.218 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.214+0000 7f24adf91700 1 -- 172.21.15.94:0/656174982 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a804c380 con 0x7f24a81407c0 2023-09-30T00:21:12.218 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.214+0000 7f24a4ff9700 1 -- 172.21.15.94:0/656174982 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2490011cb0 con 0x7f24a81407c0 2023-09-30T00:21:12.218 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.214+0000 7f24a4ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:12.218 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.215+0000 7f24a4ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:12.219 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.215+0000 7f24a4ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:12.219 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.215+0000 7f24a4ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:12.219 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.215+0000 7f24a4ff9700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f24880987b0 0x7f248809ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:12.219 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.215+0000 7f24acf8f700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f24880987b0 0x7f248809ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:12.219 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.215+0000 7f24a4ff9700 1 -- 172.21.15.94:0/656174982 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f24900c2350 con 0x7f24a81407c0 2023-09-30T00:21:12.219 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.215+0000 7f24acf8f700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f24880987b0 0x7f248809ac60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f24a0007fd0 tx=0x7f24a0001650 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:12.219 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.218+0000 7f24a4ff9700 1 -- 172.21.15.94:0/656174982 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f249008fa20 con 0x7f24a81407c0 2023-09-30T00:21:12.442 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.442+0000 7f24adf91700 1 -- 172.21.15.94:0/656174982 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f24a81395e0 con 0x7f24a81407c0 2023-09-30T00:21:12.442 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.442+0000 7f24a4ff9700 1 -- 172.21.15.94:0/656174982 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2490093040 con 0x7f24a81407c0 2023-09-30T00:21:12.443 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:12.443 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:12.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 10 mgrc shutdown 2023-09-30T00:21:12.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 1 -- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f24880987b0 msgr2=0x7f248809ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:12.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f24880987b0 0x7f248809ac60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f24a0007fd0 tx=0x7f24a0001650 comp rx=0 tx=0).stop 2023-09-30T00:21:12.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 1 -- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24a81407c0 msgr2=0x7f24a806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:12.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24a81407c0 0x7f24a806d6b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2490016000 tx=0x7f2490017000 comp rx=0 tx=0).stop 2023-09-30T00:21:12.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 1 -- 172.21.15.94:0/656174982 shutdown_connections 2023-09-30T00:21:12.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f24880987b0 0x7f248809ac60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:12.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f24a81407c0 0x7f24a806d6b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:12.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a809dbe0 0x7f24a813c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:12.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 1 --2- 172.21.15.94:0/656174982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24a809f880 0x7f24a806d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:12.446 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 1 -- 172.21.15.94:0/656174982 >> 172.21.15.94:0/656174982 conn(0x7f24a806ad20 msgr2=0x7f24a80a45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:12.446 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 1 -- 172.21.15.94:0/656174982 shutdown_connections 2023-09-30T00:21:12.446 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:12.444+0000 7f24adf91700 1 -- 172.21.15.94:0/656174982 wait complete. 2023-09-30T00:21:15.457 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:15.581 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.581+0000 7f66327bc700 1 Processor -- start 2023-09-30T00:21:15.582 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.581+0000 7f66327bc700 1 -- start start 2023-09-30T00:21:15.584 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66327bc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f662c0b6e50 0x7f662c0a1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:15.584 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66327bc700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f662c13fe20 0x7f662c140260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:15.584 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66327bc700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662c0b7220 con 0x7f662c0a2140 2023-09-30T00:21:15.584 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66327bc700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f662c1407a0 con 0x7f662c0b6e50 2023-09-30T00:21:15.584 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66327bc700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f662c0a36c0 con 0x7f662c13fe20 2023-09-30T00:21:15.584 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66317ba700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f662c0b6e50 0x7f662c0a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:15.584 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66317ba700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f662c0b6e50 0x7f662c0a1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51458/0 (socket says 172.21.15.94:51458) 2023-09-30T00:21:15.585 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66317ba700 1 -- 172.21.15.94:0/1579152912 learned_addr learned my addr 172.21.15.94:0/1579152912 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:15.585 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66317ba700 1 -- 172.21.15.94:0/1579152912 >> v2:172.21.15.121:3300/0 conn(0x7f662c13fe20 msgr2=0x7f662c140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:15.585 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66317ba700 1 --2- 172.21.15.94:0/1579152912 >> v2:172.21.15.121:3300/0 conn(0x7f662c13fe20 0x7f662c140260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:15.585 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66317ba700 1 -- 172.21.15.94:0/1579152912 >> v1:172.21.15.121:6789/0 conn(0x7f662c0a2140 legacy=0x7f662c0a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:15.585 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66317ba700 1 -- 172.21.15.94:0/1579152912 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c0a2870 con 0x7f662c0b6e50 2023-09-30T00:21:15.585 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.582+0000 7f66317ba700 1 --2- 172.21.15.94:0/1579152912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f662c0b6e50 0x7f662c0a1080 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f662c0a0d50 tx=0x7f6628009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=317cd90bb12855e2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:15.585 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.583+0000 7f6623fff700 1 -- 172.21.15.94:0/1579152912 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662800b820 con 0x7f662c0b6e50 2023-09-30T00:21:15.585 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.583+0000 7f6623fff700 1 -- 172.21.15.94:0/1579152912 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662800b980 con 0x7f662c0b6e50 2023-09-30T00:21:15.585 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.583+0000 7f6623fff700 1 -- 172.21.15.94:0/1579152912 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662800bd20 con 0x7f662c0b6e50 2023-09-30T00:21:15.585 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.583+0000 7f66327bc700 1 -- 172.21.15.94:0/1579152912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f662c0b6e50 msgr2=0x7f662c0a1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:15.585 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.583+0000 7f66327bc700 1 --2- 172.21.15.94:0/1579152912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f662c0b6e50 0x7f662c0a1080 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f662c0a0d50 tx=0x7f6628009250 comp rx=0 tx=0).stop 2023-09-30T00:21:15.585 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.583+0000 7f66327bc700 1 -- 172.21.15.94:0/1579152912 shutdown_connections 2023-09-30T00:21:15.586 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.583+0000 7f66327bc700 1 --2- 172.21.15.94:0/1579152912 >> v2:172.21.15.121:3300/0 conn(0x7f662c13fe20 0x7f662c140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:15.586 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.583+0000 7f66327bc700 1 --2- 172.21.15.94:0/1579152912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f662c0b6e50 0x7f662c0a1080 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:15.586 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.583+0000 7f66327bc700 1 -- 172.21.15.94:0/1579152912 >> 172.21.15.94:0/1579152912 conn(0x7f662c0702f0 msgr2=0x7f662c0706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:15.586 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.583+0000 7f66327bc700 1 -- 172.21.15.94:0/1579152912 shutdown_connections 2023-09-30T00:21:15.586 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.583+0000 7f66327bc700 1 -- 172.21.15.94:0/1579152912 wait complete. 2023-09-30T00:21:15.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.584+0000 7f66327bc700 1 Processor -- start 2023-09-30T00:21:15.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.584+0000 7f66327bc700 1 -- start start 2023-09-30T00:21:15.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f66327bc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f662c0a2140 0x7f662c0b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:15.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f66327bc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f662c0b6e50 0x7f662c0a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:15.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f66327bc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f662c13fe20 0x7f662c142c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:15.588 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f66327bc700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662c143ea0 con 0x7f662c0b6e50 2023-09-30T00:21:15.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f66327bc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662c0a5ae0 con 0x7f662c0a2140 2023-09-30T00:21:15.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f66327bc700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662c0a5da0 con 0x7f662c13fe20 2023-09-30T00:21:15.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f6630fb9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f662c0b6e50 0x7f662c0a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:15.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f6630fb9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f662c0b6e50 0x7f662c0a4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51468/0 (socket says 172.21.15.94:51468) 2023-09-30T00:21:15.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f6631fbb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f662c13fe20 0x7f662c142c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:15.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f6630fb9700 1 -- 172.21.15.94:0/2767055106 learned_addr learned my addr 172.21.15.94:0/2767055106 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:15.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f66317ba700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f662c0a2140 0x7f662c0b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:15.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f6630fb9700 1 -- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f662c13fe20 msgr2=0x7f662c142c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:15.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f6630fb9700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f662c13fe20 0x7f662c142c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:15.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f6630fb9700 1 -- 172.21.15.94:0/2767055106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f662c0a2140 msgr2=0x7f662c0b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:15.589 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f6630fb9700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f662c0a2140 0x7f662c0b5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:15.590 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f6630fb9700 1 -- 172.21.15.94:0/2767055106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66280094f0 con 0x7f662c0b6e50 2023-09-30T00:21:15.590 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f6631fbb700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f662c13fe20 0x7f662c142c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:15.590 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.585+0000 7f6630fb9700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f662c0b6e50 0x7f662c0a4470 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f661c00b600 tx=0x7f661c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:15.590 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.586+0000 7f66217fa700 1 -- 172.21.15.94:0/2767055106 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c00f510 con 0x7f662c0b6e50 2023-09-30T00:21:15.590 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.586+0000 7f66217fa700 1 -- 172.21.15.94:0/2767055106 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661c00f690 con 0x7f662c0b6e50 2023-09-30T00:21:15.590 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.586+0000 7f66327bc700 1 -- 172.21.15.94:0/2767055106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662c0a3d90 con 0x7f662c0b6e50 2023-09-30T00:21:15.590 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.586+0000 7f66217fa700 1 -- 172.21.15.94:0/2767055106 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c00fa50 con 0x7f662c0b6e50 2023-09-30T00:21:15.590 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.586+0000 7f66327bc700 1 -- 172.21.15.94:0/2767055106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662c13f860 con 0x7f662c0b6e50 2023-09-30T00:21:15.590 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.587+0000 7f66327bc700 1 -- 172.21.15.94:0/2767055106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f662c0a3fb0 con 0x7f662c0b6e50 2023-09-30T00:21:15.590 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.588+0000 7f66217fa700 1 -- 172.21.15.94:0/2767055106 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f661c01a400 con 0x7f662c0b6e50 2023-09-30T00:21:15.590 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.588+0000 7f66217fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:15.590 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.588+0000 7f66217fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:15.591 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.588+0000 7f66217fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:15.591 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.588+0000 7f66217fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:15.591 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.588+0000 7f66217fa700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6610098780 0x7f661009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:15.591 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.588+0000 7f66317ba700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6610098780 0x7f661009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:15.591 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.588+0000 7f66217fa700 1 -- 172.21.15.94:0/2767055106 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f661c0bfe10 con 0x7f662c0b6e50 2023-09-30T00:21:15.591 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.588+0000 7f66317ba700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6610098780 0x7f661009ac30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f662c0a0d50 tx=0x7f6628016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:15.592 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.592+0000 7f66217fa700 1 -- 172.21.15.94:0/2767055106 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f662c0a3fb0 con 0x7f662c0b6e50 2023-09-30T00:21:15.814 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.814+0000 7f66327bc700 1 -- 172.21.15.94:0/2767055106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f662c0a7d50 con 0x7f662c0b6e50 2023-09-30T00:21:15.815 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.814+0000 7f66217fa700 1 -- 172.21.15.94:0/2767055106 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f661c08d4e0 con 0x7f662c0b6e50 2023-09-30T00:21:15.815 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:15.815 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:15.817 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 10 mgrc shutdown 2023-09-30T00:21:15.817 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 1 -- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6610098780 msgr2=0x7f661009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:15.817 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6610098780 0x7f661009ac30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f662c0a0d50 tx=0x7f6628016000 comp rx=0 tx=0).stop 2023-09-30T00:21:15.817 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 1 -- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f662c0b6e50 msgr2=0x7f662c0a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:15.817 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f662c0b6e50 0x7f662c0a4470 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f661c00b600 tx=0x7f661c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:21:15.817 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 1 -- 172.21.15.94:0/2767055106 shutdown_connections 2023-09-30T00:21:15.818 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6610098780 0x7f661009ac30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:15.818 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f662c13fe20 0x7f662c142c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:15.818 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f662c0a2140 0x7f662c0b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:15.818 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 1 --2- 172.21.15.94:0/2767055106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f662c0b6e50 0x7f662c0a4470 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:15.818 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 1 -- 172.21.15.94:0/2767055106 >> 172.21.15.94:0/2767055106 conn(0x7f662c0702f0 msgr2=0x7f662c067ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:15.818 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 1 -- 172.21.15.94:0/2767055106 shutdown_connections 2023-09-30T00:21:15.818 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:15.816+0000 7f66327bc700 1 -- 172.21.15.94:0/2767055106 wait complete. 2023-09-30T00:21:18.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:18.954 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.953+0000 7f4bbf59e700 1 Processor -- start 2023-09-30T00:21:18.954 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.953+0000 7f4bbf59e700 1 -- start start 2023-09-30T00:21:18.956 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbf59e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc0140530 0x7f4bc00a2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:18.956 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbf59e700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc00b7220 con 0x7f4bc00b6e50 2023-09-30T00:21:18.956 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbf59e700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc00a14a0 con 0x7f4bc00a1080 2023-09-30T00:21:18.956 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbf59e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc00a36c0 con 0x7f4bc0140530 2023-09-30T00:21:18.956 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbe59c700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4bc00b6e50 0x7f4bc00a2fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50520/0 (socket says 172.21.15.94:50520) 2023-09-30T00:21:18.956 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbe59c700 1 -- 172.21.15.94:0/1259163229 learned_addr learned my addr 172.21.15.94:0/1259163229 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:21:18.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbed9d700 1 --2- 172.21.15.94:0/1259163229 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc0140530 0x7f4bc00a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:18.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbdd9b700 1 -- 172.21.15.94:0/1259163229 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559010227 0 0) 0x7f4bc00b7220 con 0x7f4bc00b6e50 2023-09-30T00:21:18.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbdd9b700 1 -- 172.21.15.94:0/1259163229 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bac003570 con 0x7f4bc00b6e50 2023-09-30T00:21:18.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbed9d700 1 -- 172.21.15.94:0/1259163229 >> v1:172.21.15.121:6789/0 conn(0x7f4bc00a1080 legacy=0x7f4bc013fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:18.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbed9d700 1 -- 172.21.15.94:0/1259163229 >> v1:172.21.15.94:6789/0 conn(0x7f4bc00b6e50 legacy=0x7f4bc00a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:18.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbed9d700 1 -- 172.21.15.94:0/1259163229 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc00a2870 con 0x7f4bc0140530 2023-09-30T00:21:18.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.954+0000 7f4bbed9d700 1 --2- 172.21.15.94:0/1259163229 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc0140530 0x7f4bc00a2140 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009e90 tx=0x7f4bb8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d2a664120978031 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:18.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.955+0000 7f4bbdd9b700 1 -- 172.21.15.94:0/1259163229 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb800e070 con 0x7f4bc0140530 2023-09-30T00:21:18.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.955+0000 7f4bbdd9b700 1 -- 172.21.15.94:0/1259163229 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb800bc30 con 0x7f4bc0140530 2023-09-30T00:21:18.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.955+0000 7f4bbdd9b700 1 -- 172.21.15.94:0/1259163229 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8012430 con 0x7f4bc0140530 2023-09-30T00:21:18.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.955+0000 7f4bbf59e700 1 -- 172.21.15.94:0/1259163229 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc0140530 msgr2=0x7f4bc00a2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:18.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.955+0000 7f4bbf59e700 1 --2- 172.21.15.94:0/1259163229 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc0140530 0x7f4bc00a2140 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009e90 tx=0x7f4bb8009780 comp rx=0 tx=0).stop 2023-09-30T00:21:18.958 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.955+0000 7f4bbf59e700 1 -- 172.21.15.94:0/1259163229 shutdown_connections 2023-09-30T00:21:18.958 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.955+0000 7f4bbf59e700 1 --2- 172.21.15.94:0/1259163229 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc0140530 0x7f4bc00a2140 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:18.958 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.955+0000 7f4bbf59e700 1 -- 172.21.15.94:0/1259163229 >> 172.21.15.94:0/1259163229 conn(0x7f4bc00702f0 msgr2=0x7f4bc00706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:18.958 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.955+0000 7f4bbf59e700 1 -- 172.21.15.94:0/1259163229 shutdown_connections 2023-09-30T00:21:18.958 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.955+0000 7f4bbf59e700 1 -- 172.21.15.94:0/1259163229 wait complete. 2023-09-30T00:21:18.960 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.956+0000 7f4bbf59e700 1 Processor -- start 2023-09-30T00:21:18.961 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.956+0000 7f4bbf59e700 1 -- start start 2023-09-30T00:21:18.961 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bbf59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bc00a1080 0x7f4bc00a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:18.961 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bbf59e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc00b6e50 0x7f4bc00a3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:18.961 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bbf59e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bc0140530 0x7f4bc01424f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:18.961 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bbf59e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc00a5be0 con 0x7f4bc0140530 2023-09-30T00:21:18.961 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bbf59e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc00a44a0 con 0x7f4bc00a1080 2023-09-30T00:21:18.961 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bbf59e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc00a6900 con 0x7f4bc00b6e50 2023-09-30T00:21:18.961 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bb7fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc00b6e50 0x7f4bc00a3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:18.961 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bbed9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bc0140530 0x7f4bc01424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:18.961 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bb7fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc00b6e50 0x7f4bc00a3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44566/0 (socket says 172.21.15.94:44566) 2023-09-30T00:21:18.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bb7fff700 1 -- 172.21.15.94:0/3841054086 learned_addr learned my addr 172.21.15.94:0/3841054086 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:18.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bbed9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bc0140530 0x7f4bc01424f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51480/0 (socket says 172.21.15.94:51480) 2023-09-30T00:21:18.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bbe59c700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bc00a1080 0x7f4bc00a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:18.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bb7fff700 1 -- 172.21.15.94:0/3841054086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bc00a1080 msgr2=0x7f4bc00a63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:18.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bb7fff700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bc00a1080 0x7f4bc00a63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:18.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bb7fff700 1 -- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bc0140530 msgr2=0x7f4bc01424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:18.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bb7fff700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bc0140530 0x7f4bc01424f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:18.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bb7fff700 1 -- 172.21.15.94:0/3841054086 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8009a20 con 0x7f4bc00b6e50 2023-09-30T00:21:18.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bbed9d700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bc0140530 0x7f4bc01424f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:18.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bb7fff700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc00b6e50 0x7f4bc00a3af0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800c790 tx=0x7f4ba8007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:18.962 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bb67fc700 1 -- 172.21.15.94:0/3841054086 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8011590 con 0x7f4bc00b6e50 2023-09-30T00:21:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bb67fc700 1 -- 172.21.15.94:0/3841054086 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba80116f0 con 0x7f4bc00b6e50 2023-09-30T00:21:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bbf59e700 1 -- 172.21.15.94:0/3841054086 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc009af60 con 0x7f4bc00b6e50 2023-09-30T00:21:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.957+0000 7f4bb67fc700 1 -- 172.21.15.94:0/3841054086 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8011ab0 con 0x7f4bc00b6e50 2023-09-30T00:21:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.958+0000 7f4bbf59e700 1 -- 172.21.15.94:0/3841054086 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc01441c0 con 0x7f4bc00b6e50 2023-09-30T00:21:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.958+0000 7f4bbf59e700 1 -- 172.21.15.94:0/3841054086 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc00a78e0 con 0x7f4bc00b6e50 2023-09-30T00:21:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.959+0000 7f4bb67fc700 1 -- 172.21.15.94:0/3841054086 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f4ba801b3b0 con 0x7f4bc00b6e50 2023-09-30T00:21:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.959+0000 7f4bb67fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.960+0000 7f4bb67fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.960+0000 7f4bb67fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.960+0000 7f4bb67fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.960+0000 7f4bb67fc700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4ba0098780 0x7f4ba009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.960+0000 7f4bb67fc700 1 -- 172.21.15.94:0/3841054086 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f4ba80bf870 con 0x7f4bc00b6e50 2023-09-30T00:21:18.964 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.960+0000 7f4bbe59c700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4ba0098780 0x7f4ba009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:18.964 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.960+0000 7f4bbe59c700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4ba0098780 0x7f4ba009ac30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0007fd0 tx=0x7f4bb0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:18.964 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:18.964+0000 7f4bb67fc700 1 -- 172.21.15.94:0/3841054086 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ba808cf40 con 0x7f4bc00b6e50 2023-09-30T00:21:19.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.189+0000 7f4bbf59e700 1 -- 172.21.15.94:0/3841054086 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4bc0066e50 con 0x7f4bc00b6e50 2023-09-30T00:21:19.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.189+0000 7f4bb67fc700 1 -- 172.21.15.94:0/3841054086 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4ba8017080 con 0x7f4bc00b6e50 2023-09-30T00:21:19.190 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:19.190 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.191+0000 7f4bbf59e700 10 mgrc shutdown 2023-09-30T00:21:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.191+0000 7f4bbf59e700 1 -- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4ba0098780 msgr2=0x7f4ba009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.191+0000 7f4bbf59e700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4ba0098780 0x7f4ba009ac30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0007fd0 tx=0x7f4bb0009040 comp rx=0 tx=0).stop 2023-09-30T00:21:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.191+0000 7f4bbf59e700 1 -- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc00b6e50 msgr2=0x7f4bc00a3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.191+0000 7f4bbf59e700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc00b6e50 0x7f4bc00a3af0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800c790 tx=0x7f4ba8007f60 comp rx=0 tx=0).stop 2023-09-30T00:21:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.191+0000 7f4bbf59e700 1 -- 172.21.15.94:0/3841054086 shutdown_connections 2023-09-30T00:21:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.192+0000 7f4bbf59e700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4ba0098780 0x7f4ba009ac30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.192+0000 7f4bbf59e700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bc0140530 0x7f4bc01424f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.192+0000 7f4bbf59e700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bc00a1080 0x7f4bc00a63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.192+0000 7f4bbf59e700 1 --2- 172.21.15.94:0/3841054086 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bc00b6e50 0x7f4bc00a3af0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.192+0000 7f4bbf59e700 1 -- 172.21.15.94:0/3841054086 >> 172.21.15.94:0/3841054086 conn(0x7f4bc00702f0 msgr2=0x7f4bc00a2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.192+0000 7f4bbf59e700 1 -- 172.21.15.94:0/3841054086 shutdown_connections 2023-09-30T00:21:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:19.192+0000 7f4bbf59e700 1 -- 172.21.15.94:0/3841054086 wait complete. 2023-09-30T00:21:22.204 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:22.331 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.331+0000 7fa77bba1700 1 Processor -- start 2023-09-30T00:21:22.332 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.331+0000 7fa77bba1700 1 -- start start 2023-09-30T00:21:22.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77bba1700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa77409ca70 0x7fa77409ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:22.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77bba1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77406ae70 con 0x7fa77406aaa0 2023-09-30T00:21:22.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77bba1700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7740b48c0 con 0x7fa7740a3ff0 2023-09-30T00:21:22.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77bba1700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7740a45c0 con 0x7fa77409ca70 2023-09-30T00:21:22.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77993d700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa77406aaa0 0x7fa7740b3ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50526/0 (socket says 172.21.15.94:50526) 2023-09-30T00:21:22.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77993d700 1 -- 172.21.15.94:0/945490583 learned_addr learned my addr 172.21.15.94:0/945490583 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:21:22.334 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77a13e700 1 --2- 172.21.15.94:0/945490583 >> v2:172.21.15.94:3300/0 conn(0x7fa77409ca70 0x7fa77409ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77893b700 1 -- 172.21.15.94:0/945490583 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250924451 0 0) 0x7fa77406ae70 con 0x7fa77406aaa0 2023-09-30T00:21:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77893b700 1 -- 172.21.15.94:0/945490583 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa760003570 con 0x7fa77406aaa0 2023-09-30T00:21:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77a13e700 1 -- 172.21.15.94:0/945490583 >> v1:172.21.15.121:6789/0 conn(0x7fa7740a3ff0 legacy=0x7fa7740b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77893b700 1 -- 172.21.15.94:0/945490583 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32141887 0 0) 0x7fa7740b48c0 con 0x7fa7740a3ff0 2023-09-30T00:21:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77a13e700 1 -- 172.21.15.94:0/945490583 >> v1:172.21.15.94:6789/0 conn(0x7fa77406aaa0 legacy=0x7fa7740b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77a13e700 1 -- 172.21.15.94:0/945490583 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7740a4880 con 0x7fa77409ca70 2023-09-30T00:21:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77a13e700 1 --2- 172.21.15.94:0/945490583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa77409ca70 0x7fa77409ce90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009e60 tx=0x7fa76c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d3ee2ac2253ea9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.332+0000 7fa77893b700 1 -- 172.21.15.94:0/945490583 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c00e070 con 0x7fa77409ca70 2023-09-30T00:21:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.333+0000 7fa77893b700 1 -- 172.21.15.94:0/945490583 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c00bb60 con 0x7fa77409ca70 2023-09-30T00:21:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.333+0000 7fa77893b700 1 -- 172.21.15.94:0/945490583 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c012430 con 0x7fa77409ca70 2023-09-30T00:21:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.333+0000 7fa77bba1700 1 -- 172.21.15.94:0/945490583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa77409ca70 msgr2=0x7fa77409ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.333+0000 7fa77bba1700 1 --2- 172.21.15.94:0/945490583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa77409ca70 0x7fa77409ce90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009e60 tx=0x7fa76c009780 comp rx=0 tx=0).stop 2023-09-30T00:21:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.333+0000 7fa77bba1700 1 -- 172.21.15.94:0/945490583 shutdown_connections 2023-09-30T00:21:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.333+0000 7fa77bba1700 1 --2- 172.21.15.94:0/945490583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa77409ca70 0x7fa77409ce90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.333+0000 7fa77bba1700 1 -- 172.21.15.94:0/945490583 >> 172.21.15.94:0/945490583 conn(0x7fa774068730 msgr2=0x7fa774068b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.333+0000 7fa77bba1700 1 -- 172.21.15.94:0/945490583 shutdown_connections 2023-09-30T00:21:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.333+0000 7fa77bba1700 1 -- 172.21.15.94:0/945490583 wait complete. 2023-09-30T00:21:22.339 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.334+0000 7fa77bba1700 1 Processor -- start 2023-09-30T00:21:22.339 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.334+0000 7fa77bba1700 1 -- start start 2023-09-30T00:21:22.339 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77bba1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa77406aaa0 0x7fa7740a2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:22.339 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77bba1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa77409ca70 0x7fa77413a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:22.339 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77bba1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa7740a3ff0 0x7fa77413aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:22.339 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77bba1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa774131c20 con 0x7fa77409ca70 2023-09-30T00:21:22.339 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77bba1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77413b3e0 con 0x7fa77406aaa0 2023-09-30T00:21:22.339 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77bba1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa774131d90 con 0x7fa7740a3ff0 2023-09-30T00:21:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77913c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa77409ca70 0x7fa77413a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77a13e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa7740a3ff0 0x7fa77413aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77913c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa77409ca70 0x7fa77413a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51498/0 (socket says 172.21.15.94:51498) 2023-09-30T00:21:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77913c700 1 -- 172.21.15.94:0/2193891409 learned_addr learned my addr 172.21.15.94:0/2193891409 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77993d700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa77406aaa0 0x7fa7740a2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77913c700 1 -- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa7740a3ff0 msgr2=0x7fa77413aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77913c700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa7740a3ff0 0x7fa77413aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77913c700 1 -- 172.21.15.94:0/2193891409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa77406aaa0 msgr2=0x7fa7740a2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77913c700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa77406aaa0 0x7fa7740a2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77913c700 1 -- 172.21.15.94:0/2193891409 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c009a20 con 0x7fa77409ca70 2023-09-30T00:21:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77993d700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa77406aaa0 0x7fa7740a2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:21:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77a13e700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa7740a3ff0 0x7fa77413aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77913c700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa77409ca70 0x7fa77413a960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa76400c530 tx=0x7fa76400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa769ffb700 1 -- 172.21.15.94:0/2193891409 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa764011590 con 0x7fa77409ca70 2023-09-30T00:21:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa769ffb700 1 -- 172.21.15.94:0/2193891409 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa764011710 con 0x7fa77409ca70 2023-09-30T00:21:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa77bba1700 1 -- 172.21.15.94:0/2193891409 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77413b970 con 0x7fa77409ca70 2023-09-30T00:21:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.335+0000 7fa769ffb700 1 -- 172.21.15.94:0/2193891409 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa764011ad0 con 0x7fa77409ca70 2023-09-30T00:21:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.336+0000 7fa77bba1700 1 -- 172.21.15.94:0/2193891409 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77413ff60 con 0x7fa77409ca70 2023-09-30T00:21:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.337+0000 7fa77bba1700 1 -- 172.21.15.94:0/2193891409 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77406fbd0 con 0x7fa77409ca70 2023-09-30T00:21:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.337+0000 7fa769ffb700 1 -- 172.21.15.94:0/2193891409 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fa76401b3b0 con 0x7fa77409ca70 2023-09-30T00:21:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.338+0000 7fa769ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.338+0000 7fa769ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.338+0000 7fa769ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.338+0000 7fa769ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.338+0000 7fa769ffb700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa758098780 0x7fa75809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.338+0000 7fa769ffb700 1 -- 172.21.15.94:0/2193891409 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fa7640bfa00 con 0x7fa77409ca70 2023-09-30T00:21:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.338+0000 7fa77993d700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa758098780 0x7fa75809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.338+0000 7fa77993d700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa758098780 0x7fa75809ac30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa7700058d0 tx=0x7fa770009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:22.343 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.342+0000 7fa769ffb700 1 -- 172.21.15.94:0/2193891409 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa76408d040 con 0x7fa77409ca70 2023-09-30T00:21:22.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.563+0000 7fa77bba1700 1 -- 172.21.15.94:0/2193891409 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa77413fa80 con 0x7fa77409ca70 2023-09-30T00:21:22.564 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.563+0000 7fa769ffb700 1 -- 172.21.15.94:0/2193891409 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa764017070 con 0x7fa77409ca70 2023-09-30T00:21:22.564 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:22.564 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:22.566 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.565+0000 7fa77bba1700 10 mgrc shutdown 2023-09-30T00:21:22.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.565+0000 7fa77bba1700 1 -- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa758098780 msgr2=0x7fa75809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:22.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.565+0000 7fa77bba1700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa758098780 0x7fa75809ac30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa7700058d0 tx=0x7fa770009040 comp rx=0 tx=0).stop 2023-09-30T00:21:22.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.565+0000 7fa77bba1700 1 -- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa77409ca70 msgr2=0x7fa77413a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:22.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.565+0000 7fa77bba1700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa77409ca70 0x7fa77413a960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa76400c530 tx=0x7fa76400c840 comp rx=0 tx=0).stop 2023-09-30T00:21:22.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.566+0000 7fa77bba1700 1 -- 172.21.15.94:0/2193891409 shutdown_connections 2023-09-30T00:21:22.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.566+0000 7fa77bba1700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fa758098780 0x7fa75809ac30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:22.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.566+0000 7fa77bba1700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa7740a3ff0 0x7fa77413aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:22.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.566+0000 7fa77bba1700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa77406aaa0 0x7fa7740a2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:22.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.566+0000 7fa77bba1700 1 --2- 172.21.15.94:0/2193891409 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa77409ca70 0x7fa77413a960 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:22.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.566+0000 7fa77bba1700 1 -- 172.21.15.94:0/2193891409 >> 172.21.15.94:0/2193891409 conn(0x7fa774068730 msgr2=0x7fa7740630e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:22.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.566+0000 7fa77bba1700 1 -- 172.21.15.94:0/2193891409 shutdown_connections 2023-09-30T00:21:22.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:22.566+0000 7fa77bba1700 1 -- 172.21.15.94:0/2193891409 wait complete. 2023-09-30T00:21:25.578 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:25.703 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.702+0000 7f96285db700 1 Processor -- start 2023-09-30T00:21:25.703 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.702+0000 7f96285db700 1 -- start start 2023-09-30T00:21:25.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.703+0000 7f96285db700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9620070c50 0x7f96200a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:25.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.703+0000 7f96285db700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96200a1310 con 0x7f96200b7160 2023-09-30T00:21:25.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.703+0000 7f96285db700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9620071020 con 0x7f962013fd70 2023-09-30T00:21:25.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.703+0000 7f96285db700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96200b76a0 con 0x7f9620070c50 2023-09-30T00:21:25.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.703+0000 7f9626377700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9620070c50 0x7f96200a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:25.705 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.703+0000 7f9625b76700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f96200b7160 0x7f96200a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37078/0 (socket says 172.21.15.94:37078) 2023-09-30T00:21:25.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.703+0000 7f9625b76700 1 -- 172.21.15.94:0/2040007445 learned_addr learned my addr 172.21.15.94:0/2040007445 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:21:25.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.703+0000 7f9626377700 1 --2- 172.21.15.94:0/2040007445 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9620070c50 0x7f96200a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:39616/0 (socket says 172.21.15.94:39616) 2023-09-30T00:21:25.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.703+0000 7f9626377700 1 -- 172.21.15.94:0/2040007445 >> v1:172.21.15.121:6789/0 conn(0x7f962013fd70 legacy=0x7f962014b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:25.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.703+0000 7f9625375700 1 -- 172.21.15.94:0/2040007445 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416259302 0 0) 0x7f9614001110 con 0x7f96200b7160 2023-09-30T00:21:25.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.703+0000 7f9626377700 1 -- 172.21.15.94:0/2040007445 >> v1:172.21.15.94:6789/0 conn(0x7f96200b7160 legacy=0x7f96200a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:25.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.703+0000 7f9626377700 1 -- 172.21.15.94:0/2040007445 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96200b7940 con 0x7f9620070c50 2023-09-30T00:21:25.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.704+0000 7f9626377700 1 --2- 172.21.15.94:0/2040007445 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9620070c50 0x7f96200a0dd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9618009e90 tx=0x7f9618009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=703eff85aef8e878 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:25.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.704+0000 7f9625375700 1 -- 172.21.15.94:0/2040007445 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961800e070 con 0x7f9620070c50 2023-09-30T00:21:25.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.704+0000 7f9625375700 1 -- 172.21.15.94:0/2040007445 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961800bc50 con 0x7f9620070c50 2023-09-30T00:21:25.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.704+0000 7f9625375700 1 -- 172.21.15.94:0/2040007445 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618012430 con 0x7f9620070c50 2023-09-30T00:21:25.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.704+0000 7f96285db700 1 -- 172.21.15.94:0/2040007445 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9620070c50 msgr2=0x7f96200a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.704+0000 7f96285db700 1 --2- 172.21.15.94:0/2040007445 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9620070c50 0x7f96200a0dd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9618009e90 tx=0x7f9618009780 comp rx=0 tx=0).stop 2023-09-30T00:21:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.704+0000 7f96285db700 1 -- 172.21.15.94:0/2040007445 shutdown_connections 2023-09-30T00:21:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.704+0000 7f96285db700 1 --2- 172.21.15.94:0/2040007445 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9620070c50 0x7f96200a0dd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.704+0000 7f96285db700 1 -- 172.21.15.94:0/2040007445 >> 172.21.15.94:0/2040007445 conn(0x7f962006a090 msgr2=0x7f96200702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.704+0000 7f96285db700 1 -- 172.21.15.94:0/2040007445 shutdown_connections 2023-09-30T00:21:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.704+0000 7f96285db700 1 -- 172.21.15.94:0/2040007445 wait complete. 2023-09-30T00:21:25.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.705+0000 7f96285db700 1 Processor -- start 2023-09-30T00:21:25.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.705+0000 7f96285db700 1 -- start start 2023-09-30T00:21:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f96285db700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9620070c50 0x7f96200a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f96285db700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96200b7160 0x7f96200ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f96285db700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f962013fd70 0x7f96200aac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f96285db700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96200ab930 con 0x7f96200b7160 2023-09-30T00:21:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f96285db700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96200acb20 con 0x7f9620070c50 2023-09-30T00:21:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f96285db700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96201447a0 con 0x7f962013fd70 2023-09-30T00:21:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f9625b76700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96200b7160 0x7f96200ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f9626b78700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f962013fd70 0x7f96200aac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f9625b76700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96200b7160 0x7f96200ac5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34336/0 (socket says 172.21.15.94:34336) 2023-09-30T00:21:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f9625b76700 1 -- 172.21.15.94:0/388731081 learned_addr learned my addr 172.21.15.94:0/388731081 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f9626377700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9620070c50 0x7f96200a7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:25.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f9625b76700 1 -- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f962013fd70 msgr2=0x7f96200aac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:25.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f9625b76700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f962013fd70 0x7f96200aac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:25.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f9625b76700 1 -- 172.21.15.94:0/388731081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9620070c50 msgr2=0x7f96200a7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:25.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f9625b76700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9620070c50 0x7f96200a7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:25.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f9625b76700 1 -- 172.21.15.94:0/388731081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9618009a20 con 0x7f96200b7160 2023-09-30T00:21:25.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f9626b78700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f962013fd70 0x7f96200aac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:25.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f9625b76700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96200b7160 0x7f96200ac5e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f961400c810 tx=0x7f9614007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:25.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.706+0000 7f961e7fc700 1 -- 172.21.15.94:0/388731081 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614005b30 con 0x7f96200b7160 2023-09-30T00:21:25.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.707+0000 7f961e7fc700 1 -- 172.21.15.94:0/388731081 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9614005c90 con 0x7f96200b7160 2023-09-30T00:21:25.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.707+0000 7f961e7fc700 1 -- 172.21.15.94:0/388731081 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96140072e0 con 0x7f96200b7160 2023-09-30T00:21:25.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.707+0000 7f96285db700 1 -- 172.21.15.94:0/388731081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96200ab670 con 0x7f96200b7160 2023-09-30T00:21:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.707+0000 7f96285db700 1 -- 172.21.15.94:0/388731081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96200a7a70 con 0x7f96200b7160 2023-09-30T00:21:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.708+0000 7f96285db700 1 -- 172.21.15.94:0/388731081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9620071630 con 0x7f96200b7160 2023-09-30T00:21:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.709+0000 7f961e7fc700 1 -- 172.21.15.94:0/388731081 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f9614007690 con 0x7f96200b7160 2023-09-30T00:21:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.709+0000 7f961e7fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.709+0000 7f961e7fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.709+0000 7f961e7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.709+0000 7f961e7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.709+0000 7f961e7fc700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f960c098780 0x7f960c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.709+0000 7f9626377700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f960c098780 0x7f960c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.709+0000 7f961e7fc700 1 -- 172.21.15.94:0/388731081 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f96140bfb30 con 0x7f96200b7160 2023-09-30T00:21:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.710+0000 7f9626377700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f960c098780 0x7f960c09ac30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9618009e90 tx=0x7f9618018000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.712+0000 7f961e7fc700 1 -- 172.21.15.94:0/388731081 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f961408d170 con 0x7f96200b7160 2023-09-30T00:21:25.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.933+0000 7f96285db700 1 -- 172.21.15.94:0/388731081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f96200a1310 con 0x7f96200b7160 2023-09-30T00:21:25.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.934+0000 7f961e7fc700 1 -- 172.21.15.94:0/388731081 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f96200a1310 con 0x7f96200b7160 2023-09-30T00:21:25.935 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:25.935 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:25.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 10 mgrc shutdown 2023-09-30T00:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 1 -- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f960c098780 msgr2=0x7f960c09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f960c098780 0x7f960c09ac30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9618009e90 tx=0x7f9618018000 comp rx=0 tx=0).stop 2023-09-30T00:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 1 -- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96200b7160 msgr2=0x7f96200ac5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96200b7160 0x7f96200ac5e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f961400c810 tx=0x7f9614007c50 comp rx=0 tx=0).stop 2023-09-30T00:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 1 -- 172.21.15.94:0/388731081 shutdown_connections 2023-09-30T00:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f960c098780 0x7f960c09ac30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f962013fd70 0x7f96200aac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9620070c50 0x7f96200a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 1 --2- 172.21.15.94:0/388731081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96200b7160 0x7f96200ac5e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 1 -- 172.21.15.94:0/388731081 >> 172.21.15.94:0/388731081 conn(0x7f962006a090 msgr2=0x7f96200a8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 1 -- 172.21.15.94:0/388731081 shutdown_connections 2023-09-30T00:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:25.936+0000 7f96285db700 1 -- 172.21.15.94:0/388731081 wait complete. 2023-09-30T00:21:28.948 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:29.073 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.073+0000 7fbbaab07700 1 Processor -- start 2023-09-30T00:21:29.074 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.073+0000 7fbbaab07700 1 -- start start 2023-09-30T00:21:29.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.073+0000 7fbbaab07700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbba4144560 0x7fbba414c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:29.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbbaab07700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba4067eb0 con 0x7fbba4067ae0 2023-09-30T00:21:29.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbbaab07700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba406a230 con 0x7fbba40a51c0 2023-09-30T00:21:29.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbbaab07700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba4144ae0 con 0x7fbba4144560 2023-09-30T00:21:29.076 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbba90a4700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbba4144560 0x7fbba414c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:29.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbba90a4700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbba4144560 0x7fbba414c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34352/0 (socket says 172.21.15.94:34352) 2023-09-30T00:21:29.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbba88a3700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbba4067ae0 0x7fbba4136950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37080/0 (socket says 172.21.15.94:37080) 2023-09-30T00:21:29.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbba90a4700 1 -- 172.21.15.94:0/3699266401 learned_addr learned my addr 172.21.15.94:0/3699266401 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:29.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbba90a4700 1 -- 172.21.15.94:0/3699266401 >> v1:172.21.15.121:6789/0 conn(0x7fbba40a51c0 legacy=0x7fbba4143e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:29.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbba90a4700 1 -- 172.21.15.94:0/3699266401 >> v1:172.21.15.94:6789/0 conn(0x7fbba4067ae0 legacy=0x7fbba4136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:29.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbba90a4700 1 -- 172.21.15.94:0/3699266401 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba4137060 con 0x7fbba4144560 2023-09-30T00:21:29.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbba37fe700 1 -- 172.21.15.94:0/3699266401 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64452996 0 0) 0x7fbba4067eb0 con 0x7fbba4067ae0 2023-09-30T00:21:29.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbba90a4700 1 --2- 172.21.15.94:0/3699266401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbba4144560 0x7fbba414c1a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbba4137250 tx=0x7fbb94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6597963fefe0b24 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:29.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbba37fe700 1 -- 172.21.15.94:0/3699266401 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9400b820 con 0x7fbba4144560 2023-09-30T00:21:29.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbba37fe700 1 -- 172.21.15.94:0/3699266401 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9400b980 con 0x7fbba4144560 2023-09-30T00:21:29.077 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.074+0000 7fbba37fe700 1 -- 172.21.15.94:0/3699266401 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9400bd40 con 0x7fbba4144560 2023-09-30T00:21:29.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.075+0000 7fbbaab07700 1 -- 172.21.15.94:0/3699266401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbba4144560 msgr2=0x7fbba414c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:29.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.075+0000 7fbbaab07700 1 --2- 172.21.15.94:0/3699266401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbba4144560 0x7fbba414c1a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbba4137250 tx=0x7fbb94009250 comp rx=0 tx=0).stop 2023-09-30T00:21:29.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.075+0000 7fbbaab07700 1 -- 172.21.15.94:0/3699266401 shutdown_connections 2023-09-30T00:21:29.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.075+0000 7fbbaab07700 1 --2- 172.21.15.94:0/3699266401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbba4144560 0x7fbba414c1a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:29.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.075+0000 7fbbaab07700 1 -- 172.21.15.94:0/3699266401 >> 172.21.15.94:0/3699266401 conn(0x7fbba406cb10 msgr2=0x7fbba406cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:29.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.075+0000 7fbbaab07700 1 -- 172.21.15.94:0/3699266401 shutdown_connections 2023-09-30T00:21:29.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.075+0000 7fbbaab07700 1 -- 172.21.15.94:0/3699266401 wait complete. 2023-09-30T00:21:29.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.076+0000 7fbbaab07700 1 Processor -- start 2023-09-30T00:21:29.078 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.076+0000 7fbbaab07700 1 -- start start 2023-09-30T00:21:29.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.076+0000 7fbbaab07700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba4067ae0 0x7fbba41352d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:29.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.076+0000 7fbbaab07700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbba40a51c0 0x7fbba40a3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:29.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.076+0000 7fbbaab07700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbba4144560 0x7fbba40a4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:29.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.076+0000 7fbbaab07700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba4136220 con 0x7fbba40a51c0 2023-09-30T00:21:29.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.076+0000 7fbbaab07700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba4135ca0 con 0x7fbba4067ae0 2023-09-30T00:21:29.081 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.076+0000 7fbbaab07700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba4135f60 con 0x7fbba4144560 2023-09-30T00:21:29.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba3fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbba40a51c0 0x7fbba40a3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:29.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba90a4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbba4144560 0x7fbba40a4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:29.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba3fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbba40a51c0 0x7fbba40a3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34362/0 (socket says 172.21.15.94:34362) 2023-09-30T00:21:29.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba3fff700 1 -- 172.21.15.94:0/1172633498 learned_addr learned my addr 172.21.15.94:0/1172633498 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:29.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba88a3700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba4067ae0 0x7fbba41352d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:29.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba3fff700 1 -- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbba4144560 msgr2=0x7fbba40a4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:29.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba3fff700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbba4144560 0x7fbba40a4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:29.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba3fff700 1 -- 172.21.15.94:0/1172633498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba4067ae0 msgr2=0x7fbba41352d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:29.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba3fff700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba4067ae0 0x7fbba41352d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:29.082 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba3fff700 1 -- 172.21.15.94:0/1172633498 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb940094f0 con 0x7fbba40a51c0 2023-09-30T00:21:29.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba90a4700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbba4144560 0x7fbba40a4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:29.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba3fff700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbba40a51c0 0x7fbba40a3ce0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800c590 tx=0x7fbb9800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:29.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba0ff9700 1 -- 172.21.15.94:0/1172633498 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb98011590 con 0x7fbba40a51c0 2023-09-30T00:21:29.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba0ff9700 1 -- 172.21.15.94:0/1172633498 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb98011710 con 0x7fbba40a51c0 2023-09-30T00:21:29.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbbaab07700 1 -- 172.21.15.94:0/1172633498 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba409fe00 con 0x7fbba40a51c0 2023-09-30T00:21:29.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbba0ff9700 1 -- 172.21.15.94:0/1172633498 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb98011ad0 con 0x7fbba40a51c0 2023-09-30T00:21:29.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.077+0000 7fbbaab07700 1 -- 172.21.15.94:0/1172633498 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba4135a50 con 0x7fbba40a51c0 2023-09-30T00:21:29.083 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.078+0000 7fbbaab07700 1 -- 172.21.15.94:0/1172633498 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba40a71d0 con 0x7fbba40a51c0 2023-09-30T00:21:29.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.079+0000 7fbba0ff9700 1 -- 172.21.15.94:0/1172633498 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fbb9801b3b0 con 0x7fbba40a51c0 2023-09-30T00:21:29.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.079+0000 7fbba0ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:29.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.080+0000 7fbba0ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:29.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.080+0000 7fbba0ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:29.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.080+0000 7fbba0ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:29.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.080+0000 7fbba0ff9700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fbb90098780 0x7fbb9009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:29.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.080+0000 7fbba88a3700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fbb90098780 0x7fbb9009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:29.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.080+0000 7fbba0ff9700 1 -- 172.21.15.94:0/1172633498 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fbb980bfdb0 con 0x7fbba40a51c0 2023-09-30T00:21:29.084 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.080+0000 7fbba88a3700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fbb90098780 0x7fbb9009ac30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c001560 tx=0x7fbb9c001ae0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:29.085 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.083+0000 7fbba0ff9700 1 -- 172.21.15.94:0/1172633498 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbb9808d350 con 0x7fbba40a51c0 2023-09-30T00:21:29.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.306+0000 7fbbaab07700 1 -- 172.21.15.94:0/1172633498 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbba4071630 con 0x7fbba40a51c0 2023-09-30T00:21:29.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.306+0000 7fbba0ff9700 1 -- 172.21.15.94:0/1172633498 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbb98017070 con 0x7fbba40a51c0 2023-09-30T00:21:29.308 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:29.308 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:29.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 10 mgrc shutdown 2023-09-30T00:21:29.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 1 -- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fbb90098780 msgr2=0x7fbb9009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:29.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fbb90098780 0x7fbb9009ac30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c001560 tx=0x7fbb9c001ae0 comp rx=0 tx=0).stop 2023-09-30T00:21:29.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 1 -- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbba40a51c0 msgr2=0x7fbba40a3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:29.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbba40a51c0 0x7fbba40a3ce0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800c590 tx=0x7fbb9800c8a0 comp rx=0 tx=0).stop 2023-09-30T00:21:29.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 1 -- 172.21.15.94:0/1172633498 shutdown_connections 2023-09-30T00:21:29.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fbb90098780 0x7fbb9009ac30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:29.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fbba4144560 0x7fbba40a4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba4067ae0 0x7fbba41352d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 1 --2- 172.21.15.94:0/1172633498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbba40a51c0 0x7fbba40a3ce0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 1 -- 172.21.15.94:0/1172633498 >> 172.21.15.94:0/1172633498 conn(0x7fbba406cb10 msgr2=0x7fbba4136720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 1 -- 172.21.15.94:0/1172633498 shutdown_connections 2023-09-30T00:21:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:29.308+0000 7fbbaab07700 1 -- 172.21.15.94:0/1172633498 wait complete. 2023-09-30T00:21:32.322 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:32.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.439+0000 7f651f6e3700 1 Processor -- start 2023-09-30T00:21:32.440 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.439+0000 7f651f6e3700 1 -- start start 2023-09-30T00:21:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.439+0000 7f651f6e3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65180b7160 0x7f65180a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.439+0000 7f651f6e3700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65180a1310 con 0x7f6518070c50 2023-09-30T00:21:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.439+0000 7f651f6e3700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6518071020 con 0x7f651813fd70 2023-09-30T00:21:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.439+0000 7f651f6e3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65180b76a0 con 0x7f65180b7160 2023-09-30T00:21:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.439+0000 7f651cc7e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65180b7160 0x7f65180a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.439+0000 7f651d47f700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6518070c50 0x7f65180a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37082/0 (socket says 172.21.15.94:37082) 2023-09-30T00:21:32.444 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.439+0000 7f651d47f700 1 -- 172.21.15.94:0/3788037271 learned_addr learned my addr 172.21.15.94:0/3788037271 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:21:32.444 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.440+0000 7f6517fff700 1 -- 172.21.15.94:0/3788037271 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797278142 0 0) 0x7f6508001110 con 0x7f6518070c50 2023-09-30T00:21:32.444 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.440+0000 7f6517fff700 1 -- 172.21.15.94:0/3788037271 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6500003570 con 0x7f6518070c50 2023-09-30T00:21:32.444 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.440+0000 7f651cc7e700 1 -- 172.21.15.94:0/3788037271 >> v1:172.21.15.121:6789/0 conn(0x7f651813fd70 legacy=0x7f651814b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:32.444 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.440+0000 7f651cc7e700 1 -- 172.21.15.94:0/3788037271 >> v1:172.21.15.94:6789/0 conn(0x7f6518070c50 legacy=0x7f65180a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:32.444 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.440+0000 7f651cc7e700 1 -- 172.21.15.94:0/3788037271 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65180b7940 con 0x7f65180b7160 2023-09-30T00:21:32.444 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.440+0000 7f651cc7e700 1 --2- 172.21.15.94:0/3788037271 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65180b7160 0x7f65180a0dd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f650c009910 tx=0x7f650c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dc7180c3cdc1bc8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:32.444 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.440+0000 7f6517fff700 1 -- 172.21.15.94:0/3788037271 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c00b710 con 0x7f65180b7160 2023-09-30T00:21:32.444 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.440+0000 7f6517fff700 1 -- 172.21.15.94:0/3788037271 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c00b870 con 0x7f65180b7160 2023-09-30T00:21:32.444 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.440+0000 7f6517fff700 1 -- 172.21.15.94:0/3788037271 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c00bc30 con 0x7f65180b7160 2023-09-30T00:21:32.444 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.440+0000 7f651f6e3700 1 -- 172.21.15.94:0/3788037271 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65180b7160 msgr2=0x7f65180a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.440+0000 7f651f6e3700 1 --2- 172.21.15.94:0/3788037271 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65180b7160 0x7f65180a0dd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f650c009910 tx=0x7f650c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:21:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.441+0000 7f651f6e3700 1 -- 172.21.15.94:0/3788037271 shutdown_connections 2023-09-30T00:21:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.441+0000 7f651f6e3700 1 --2- 172.21.15.94:0/3788037271 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f65180b7160 0x7f65180a0dd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.441+0000 7f651f6e3700 1 -- 172.21.15.94:0/3788037271 >> 172.21.15.94:0/3788037271 conn(0x7f651806a090 msgr2=0x7f65180702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.441+0000 7f651f6e3700 1 -- 172.21.15.94:0/3788037271 shutdown_connections 2023-09-30T00:21:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.441+0000 7f651f6e3700 1 -- 172.21.15.94:0/3788037271 wait complete. 2023-09-30T00:21:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.441+0000 7f651f6e3700 1 Processor -- start 2023-09-30T00:21:32.445 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.442+0000 7f651f6e3700 1 -- start start 2023-09-30T00:21:32.447 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.442+0000 7f651f6e3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6518070c50 0x7f65180a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:32.447 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.442+0000 7f651f6e3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65180b7160 0x7f6518143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.442+0000 7f651f6e3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f651813fd70 0x7f65180ab5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.442+0000 7f651f6e3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65180a6df0 con 0x7f65180b7160 2023-09-30T00:21:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.442+0000 7f651f6e3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65180a6c30 con 0x7f6518070c50 2023-09-30T00:21:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.442+0000 7f651f6e3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65180a4010 con 0x7f651813fd70 2023-09-30T00:21:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.442+0000 7f651cc7e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65180b7160 0x7f6518143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.442+0000 7f651dc80700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f651813fd70 0x7f65180ab5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.442+0000 7f651cc7e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65180b7160 0x7f6518143bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34376/0 (socket says 172.21.15.94:34376) 2023-09-30T00:21:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.442+0000 7f651cc7e700 1 -- 172.21.15.94:0/2425666004 learned_addr learned my addr 172.21.15.94:0/2425666004 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.442+0000 7f651d47f700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6518070c50 0x7f65180a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.443+0000 7f651dc80700 1 -- 172.21.15.94:0/2425666004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6518070c50 msgr2=0x7f65180a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.443+0000 7f651dc80700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6518070c50 0x7f65180a66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.443+0000 7f651dc80700 1 -- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65180b7160 msgr2=0x7f6518143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.443+0000 7f651dc80700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65180b7160 0x7f6518143bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.443+0000 7f651dc80700 1 -- 172.21.15.94:0/2425666004 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650c009420 con 0x7f651813fd70 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.443+0000 7f651cc7e700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65180b7160 0x7f6518143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.443+0000 7f651dc80700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f651813fd70 0x7f65180ab5b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f651000c7c0 tx=0x7f65100015f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.443+0000 7f65157fa700 1 -- 172.21.15.94:0/2425666004 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510011590 con 0x7f651813fd70 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.443+0000 7f65157fa700 1 -- 172.21.15.94:0/2425666004 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6510011710 con 0x7f651813fd70 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.443+0000 7f651f6e3700 1 -- 172.21.15.94:0/2425666004 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6518144130 con 0x7f651813fd70 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.443+0000 7f65157fa700 1 -- 172.21.15.94:0/2425666004 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510011ad0 con 0x7f651813fd70 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.443+0000 7f651f6e3700 1 -- 172.21.15.94:0/2425666004 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6518144490 con 0x7f651813fd70 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.444+0000 7f651f6e3700 1 -- 172.21.15.94:0/2425666004 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6518071630 con 0x7f651813fd70 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.445+0000 7f65157fa700 1 -- 172.21.15.94:0/2425666004 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f6510003030 con 0x7f651813fd70 2023-09-30T00:21:32.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.445+0000 7f65157fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:32.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.445+0000 7f65157fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:32.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.445+0000 7f65157fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:32.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.445+0000 7f65157fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:32.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.445+0000 7f65157fa700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6504098780 0x7f650409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:32.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.445+0000 7f651d47f700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6504098780 0x7f650409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:32.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.445+0000 7f65157fa700 1 -- 172.21.15.94:0/2425666004 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f65100bfe50 con 0x7f651813fd70 2023-09-30T00:21:32.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.446+0000 7f651d47f700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6504098780 0x7f650409ac30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6508001270 tx=0x7f6508007bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:32.450 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.449+0000 7f65157fa700 1 -- 172.21.15.94:0/2425666004 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f651008d520 con 0x7f651813fd70 2023-09-30T00:21:32.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.672+0000 7f651f6e3700 1 -- 172.21.15.94:0/2425666004 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f65180a5180 con 0x7f651813fd70 2023-09-30T00:21:32.673 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.672+0000 7f65157fa700 1 -- 172.21.15.94:0/2425666004 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6510096070 con 0x7f651813fd70 2023-09-30T00:21:32.674 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:32.674 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:32.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 10 mgrc shutdown 2023-09-30T00:21:32.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 1 -- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6504098780 msgr2=0x7f650409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:32.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6504098780 0x7f650409ac30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6508001270 tx=0x7f6508007bf0 comp rx=0 tx=0).stop 2023-09-30T00:21:32.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 1 -- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f651813fd70 msgr2=0x7f65180ab5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:32.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f651813fd70 0x7f65180ab5b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f651000c7c0 tx=0x7f65100015f0 comp rx=0 tx=0).stop 2023-09-30T00:21:32.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 1 -- 172.21.15.94:0/2425666004 shutdown_connections 2023-09-30T00:21:32.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f6504098780 0x7f650409ac30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:32.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f651813fd70 0x7f65180ab5b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:32.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6518070c50 0x7f65180a66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:32.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 1 --2- 172.21.15.94:0/2425666004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65180b7160 0x7f6518143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:32.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 1 -- 172.21.15.94:0/2425666004 >> 172.21.15.94:0/2425666004 conn(0x7f651806a090 msgr2=0x7f65180a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:32.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 1 -- 172.21.15.94:0/2425666004 shutdown_connections 2023-09-30T00:21:32.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:32.674+0000 7f651f6e3700 1 -- 172.21.15.94:0/2425666004 wait complete. 2023-09-30T00:21:35.687 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:35.813 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.811+0000 7fb67714b700 1 Processor -- start 2023-09-30T00:21:35.813 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.811+0000 7fb67714b700 1 -- start start 2023-09-30T00:21:35.815 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.812+0000 7fb67714b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb6700b6e50 0x7fb6700a1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:35.815 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.812+0000 7fb67714b700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6700b7220 con 0x7fb6700a2140 2023-09-30T00:21:35.815 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.812+0000 7fb67714b700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6700a2870 con 0x7fb67013fe20 2023-09-30T00:21:35.815 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.812+0000 7fb67714b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6700a36c0 con 0x7fb6700b6e50 2023-09-30T00:21:35.815 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.812+0000 7fb676149700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb6700b6e50 0x7fb6700a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:35.815 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.812+0000 7fb675948700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb6700a2140 0x7fb6700a2fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38342/0 (socket says 172.21.15.94:38342) 2023-09-30T00:21:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.812+0000 7fb675948700 1 -- 172.21.15.94:0/2952183365 learned_addr learned my addr 172.21.15.94:0/2952183365 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:21:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.812+0000 7fb676149700 1 -- 172.21.15.94:0/2952183365 >> v1:172.21.15.121:6789/0 conn(0x7fb67013fe20 legacy=0x7fb67014b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.812+0000 7fb676149700 1 -- 172.21.15.94:0/2952183365 >> v1:172.21.15.94:6789/0 conn(0x7fb6700a2140 legacy=0x7fb6700a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.812+0000 7fb675147700 1 -- 172.21.15.94:0/2952183365 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388746096 0 0) 0x7fb6700b7220 con 0x7fb6700a2140 2023-09-30T00:21:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.812+0000 7fb676149700 1 -- 172.21.15.94:0/2952183365 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6700a2b30 con 0x7fb6700b6e50 2023-09-30T00:21:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.812+0000 7fb676149700 1 --2- 172.21.15.94:0/2952183365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6700b6e50 0x7fb6700a1080 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0098e0 tx=0x7fb66c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=626f7ac0800baa5c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.813+0000 7fb675147700 1 -- 172.21.15.94:0/2952183365 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c009e80 con 0x7fb6700b6e50 2023-09-30T00:21:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.813+0000 7fb675147700 1 -- 172.21.15.94:0/2952183365 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c00b820 con 0x7fb6700b6e50 2023-09-30T00:21:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.813+0000 7fb675147700 1 -- 172.21.15.94:0/2952183365 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c00bba0 con 0x7fb6700b6e50 2023-09-30T00:21:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.813+0000 7fb67714b700 1 -- 172.21.15.94:0/2952183365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6700b6e50 msgr2=0x7fb6700a1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.813+0000 7fb67714b700 1 --2- 172.21.15.94:0/2952183365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6700b6e50 0x7fb6700a1080 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0098e0 tx=0x7fb66c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:21:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.813+0000 7fb67714b700 1 -- 172.21.15.94:0/2952183365 shutdown_connections 2023-09-30T00:21:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.813+0000 7fb67714b700 1 --2- 172.21.15.94:0/2952183365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6700b6e50 0x7fb6700a1080 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.813+0000 7fb67714b700 1 -- 172.21.15.94:0/2952183365 >> 172.21.15.94:0/2952183365 conn(0x7fb6700702f0 msgr2=0x7fb6700706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.813+0000 7fb67714b700 1 -- 172.21.15.94:0/2952183365 shutdown_connections 2023-09-30T00:21:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.813+0000 7fb67714b700 1 -- 172.21.15.94:0/2952183365 wait complete. 2023-09-30T00:21:35.820 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.814+0000 7fb67714b700 1 Processor -- start 2023-09-30T00:21:35.820 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.814+0000 7fb67714b700 1 -- start start 2023-09-30T00:21:35.820 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb67714b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6700a2140 0x7fb670145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:35.820 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb67714b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6700b6e50 0x7fb670153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:35.820 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb67714b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67013fe20 0x7fb67013a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:35.820 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb67714b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb670143460 con 0x7fb6700a2140 2023-09-30T00:21:35.820 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb67714b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb670144360 con 0x7fb67013fe20 2023-09-30T00:21:35.820 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb67714b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb670143bb0 con 0x7fb6700b6e50 2023-09-30T00:21:35.820 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb676149700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6700a2140 0x7fb670145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb676149700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6700a2140 0x7fb670145790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59364/0 (socket says 172.21.15.94:59364) 2023-09-30T00:21:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb676149700 1 -- 172.21.15.94:0/2959197762 learned_addr learned my addr 172.21.15.94:0/2959197762 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb675948700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6700b6e50 0x7fb670153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb676149700 1 -- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6700b6e50 msgr2=0x7fb670153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb676149700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6700b6e50 0x7fb670153b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb67694a700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67013fe20 0x7fb67013a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb676149700 1 -- 172.21.15.94:0/2959197762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67013fe20 msgr2=0x7fb67013a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb676149700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67013fe20 0x7fb67013a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb676149700 1 -- 172.21.15.94:0/2959197762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c009420 con 0x7fb6700a2140 2023-09-30T00:21:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb676149700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6700a2140 0x7fb670145790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb670071280 tx=0x7fb66c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb6667fc700 1 -- 172.21.15.94:0/2959197762 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c00ba60 con 0x7fb6700a2140 2023-09-30T00:21:35.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb6667fc700 1 -- 172.21.15.94:0/2959197762 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c020070 con 0x7fb6700a2140 2023-09-30T00:21:35.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb6667fc700 1 -- 172.21.15.94:0/2959197762 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c01b6a0 con 0x7fb6700a2140 2023-09-30T00:21:35.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb67714b700 1 -- 172.21.15.94:0/2959197762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb670154310 con 0x7fb6700a2140 2023-09-30T00:21:35.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.815+0000 7fb67714b700 1 -- 172.21.15.94:0/2959197762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6700a68e0 con 0x7fb6700a2140 2023-09-30T00:21:35.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.816+0000 7fb67714b700 1 -- 172.21.15.94:0/2959197762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb670066e50 con 0x7fb6700a2140 2023-09-30T00:21:35.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.817+0000 7fb6667fc700 1 -- 172.21.15.94:0/2959197762 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb66c022030 con 0x7fb6700a2140 2023-09-30T00:21:35.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.817+0000 7fb6667fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:35.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.818+0000 7fb6667fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:35.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.818+0000 7fb6667fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:35.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.818+0000 7fb6667fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:35.822 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.818+0000 7fb6667fc700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb654098780 0x7fb65409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:35.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.818+0000 7fb675948700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb654098780 0x7fb65409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:35.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.818+0000 7fb6667fc700 1 -- 172.21.15.94:0/2959197762 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fb66c0c2770 con 0x7fb6700a2140 2023-09-30T00:21:35.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.818+0000 7fb675948700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb654098780 0x7fb65409ac30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb6700709b0 tx=0x7fb660007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:35.823 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:35.821+0000 7fb6667fc700 1 -- 172.21.15.94:0/2959197762 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb66c08fe40 con 0x7fb6700a2140 2023-09-30T00:21:36.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.046+0000 7fb67714b700 1 -- 172.21.15.94:0/2959197762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb670066e50 con 0x7fb6700a2140 2023-09-30T00:21:36.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.046+0000 7fb6667fc700 1 -- 172.21.15.94:0/2959197762 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb66c093460 con 0x7fb6700a2140 2023-09-30T00:21:36.048 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:36.048 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:36.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.048+0000 7fb67714b700 10 mgrc shutdown 2023-09-30T00:21:36.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.048+0000 7fb67714b700 1 -- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb654098780 msgr2=0x7fb65409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:36.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.048+0000 7fb67714b700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb654098780 0x7fb65409ac30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb6700709b0 tx=0x7fb660007d30 comp rx=0 tx=0).stop 2023-09-30T00:21:36.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.048+0000 7fb67714b700 1 -- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6700a2140 msgr2=0x7fb670145790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:36.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.048+0000 7fb67714b700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6700a2140 0x7fb670145790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb670071280 tx=0x7fb66c016000 comp rx=0 tx=0).stop 2023-09-30T00:21:36.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.048+0000 7fb67714b700 1 -- 172.21.15.94:0/2959197762 shutdown_connections 2023-09-30T00:21:36.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.048+0000 7fb67714b700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb654098780 0x7fb65409ac30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:36.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.048+0000 7fb67714b700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67013fe20 0x7fb67013a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:36.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.048+0000 7fb67714b700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6700a2140 0x7fb670145790 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:36.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.048+0000 7fb67714b700 1 --2- 172.21.15.94:0/2959197762 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb6700b6e50 0x7fb670153b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:36.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.048+0000 7fb67714b700 1 -- 172.21.15.94:0/2959197762 >> 172.21.15.94:0/2959197762 conn(0x7fb6700702f0 msgr2=0x7fb6701460e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:36.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.049+0000 7fb67714b700 1 -- 172.21.15.94:0/2959197762 shutdown_connections 2023-09-30T00:21:36.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:36.049+0000 7fb67714b700 1 -- 172.21.15.94:0/2959197762 wait complete. 2023-09-30T00:21:39.062 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:39.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.185+0000 7f7baaa8f700 1 Processor -- start 2023-09-30T00:21:39.187 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.185+0000 7f7baaa8f700 1 -- start start 2023-09-30T00:21:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7baaa8f700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7ba406eab0 0x7f7ba4140b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7baaa8f700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ba40a2ef0 0x7f7ba4147e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7baaa8f700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba4141180 con 0x7f7ba409de30 2023-09-30T00:21:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7baaa8f700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba4141440 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7baaa8f700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba4141700 con 0x7f7ba406eab0 2023-09-30T00:21:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7ba902c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ba40a2ef0 0x7f7ba4147e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7ba902c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ba40a2ef0 0x7f7ba4147e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59370/0 (socket says 172.21.15.94:59370) 2023-09-30T00:21:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7ba902c700 1 -- 172.21.15.94:0/3390684670 learned_addr learned my addr 172.21.15.94:0/3390684670 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7b9bfff700 1 --2- 172.21.15.94:0/3390684670 >> v2:172.21.15.121:3300/0 conn(0x7f7ba406eab0 0x7f7ba4140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7ba902c700 1 -- 172.21.15.94:0/3390684670 >> v2:172.21.15.121:3300/0 conn(0x7f7ba406eab0 msgr2=0x7f7ba4140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7ba902c700 1 --2- 172.21.15.94:0/3390684670 >> v2:172.21.15.121:3300/0 conn(0x7f7ba406eab0 0x7f7ba4140b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7ba902c700 1 -- 172.21.15.94:0/3390684670 >> v1:172.21.15.121:6789/0 conn(0x7f7ba409de30 legacy=0x7f7ba409f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.186+0000 7f7ba902c700 1 -- 172.21.15.94:0/3390684670 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba409e200 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.187+0000 7f7ba902c700 1 --2- 172.21.15.94:0/3390684670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ba40a2ef0 0x7f7ba4147e40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7ba409ece0 tx=0x7f7b9c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938f5b5f0334695d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.187+0000 7f7b9b7fe700 1 -- 172.21.15.94:0/3390684670 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c00e070 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.187+0000 7f7b9b7fe700 1 -- 172.21.15.94:0/3390684670 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9c00bab0 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.187+0000 7f7b9b7fe700 1 -- 172.21.15.94:0/3390684670 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c0092e0 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.187+0000 7f7baaa8f700 1 -- 172.21.15.94:0/3390684670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ba40a2ef0 msgr2=0x7f7ba4147e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:39.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.187+0000 7f7baaa8f700 1 --2- 172.21.15.94:0/3390684670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ba40a2ef0 0x7f7ba4147e40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7ba409ece0 tx=0x7f7b9c009840 comp rx=0 tx=0).stop 2023-09-30T00:21:39.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.187+0000 7f7baaa8f700 1 -- 172.21.15.94:0/3390684670 shutdown_connections 2023-09-30T00:21:39.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.187+0000 7f7baaa8f700 1 --2- 172.21.15.94:0/3390684670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ba40a2ef0 0x7f7ba4147e40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:39.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.187+0000 7f7baaa8f700 1 --2- 172.21.15.94:0/3390684670 >> v2:172.21.15.121:3300/0 conn(0x7f7ba406eab0 0x7f7ba4140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:39.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.187+0000 7f7baaa8f700 1 -- 172.21.15.94:0/3390684670 >> 172.21.15.94:0/3390684670 conn(0x7f7ba40b3520 msgr2=0x7f7ba4067620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:39.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.187+0000 7f7baaa8f700 1 -- 172.21.15.94:0/3390684670 shutdown_connections 2023-09-30T00:21:39.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.187+0000 7f7baaa8f700 1 -- 172.21.15.94:0/3390684670 wait complete. 2023-09-30T00:21:39.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.188+0000 7f7baaa8f700 1 Processor -- start 2023-09-30T00:21:39.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.188+0000 7f7baaa8f700 1 -- start start 2023-09-30T00:21:39.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7baaa8f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ba406eab0 0x7f7ba40981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:39.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7baaa8f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7ba409de30 0x7f7ba4140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:39.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7baaa8f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ba40a2ef0 0x7f7ba413b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:39.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7baaa8f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba414bff0 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7baaa8f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba40993e0 con 0x7f7ba406eab0 2023-09-30T00:21:39.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7baaa8f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba40996a0 con 0x7f7ba409de30 2023-09-30T00:21:39.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7ba902c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ba40a2ef0 0x7f7ba413b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:39.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7b9bfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7ba409de30 0x7f7ba4140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:39.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7ba902c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ba40a2ef0 0x7f7ba413b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59382/0 (socket says 172.21.15.94:59382) 2023-09-30T00:21:39.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7b9bfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7ba409de30 0x7f7ba4140540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:53236/0 (socket says 172.21.15.94:53236) 2023-09-30T00:21:39.194 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7ba902c700 1 -- 172.21.15.94:0/4237824778 learned_addr learned my addr 172.21.15.94:0/4237824778 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7ba882b700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ba406eab0 0x7f7ba40981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7ba902c700 1 -- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7ba409de30 msgr2=0x7f7ba4140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7ba902c700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7ba409de30 0x7f7ba4140540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7ba902c700 1 -- 172.21.15.94:0/4237824778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ba406eab0 msgr2=0x7f7ba40981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7ba902c700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ba406eab0 0x7f7ba40981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7ba902c700 1 -- 172.21.15.94:0/4237824778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c009ae0 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.189+0000 7f7b9bfff700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7ba409de30 0x7f7ba4140540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.190+0000 7f7ba902c700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ba40a2ef0 0x7f7ba413b760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7ba40698b0 tx=0x7f7b9c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.190+0000 7f7b98ff9700 1 -- 172.21.15.94:0/4237824778 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c0032f0 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.190+0000 7f7b98ff9700 1 -- 172.21.15.94:0/4237824778 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9c008ce0 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.190+0000 7f7baaa8f700 1 -- 172.21.15.94:0/4237824778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba4099960 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.190+0000 7f7b98ff9700 1 -- 172.21.15.94:0/4237824778 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c0266b0 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.190+0000 7f7baaa8f700 1 -- 172.21.15.94:0/4237824778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba406cab0 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.191+0000 7f7baaa8f700 1 -- 172.21.15.94:0/4237824778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba4071630 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.192+0000 7f7b98ff9700 1 -- 172.21.15.94:0/4237824778 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f7b9c003040 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.192+0000 7f7b98ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.192+0000 7f7b98ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.192+0000 7f7b98ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.192+0000 7f7b98ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.192+0000 7f7b98ff9700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b90098730 0x7f7b9009abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.192+0000 7f7b98ff9700 1 -- 172.21.15.94:0/4237824778 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f7b9c0c1d90 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.192+0000 7f7ba882b700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b90098730 0x7f7b9009abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.193+0000 7f7ba882b700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b90098730 0x7f7b9009abe0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0003a80 tx=0x7f7ba0007ec0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.196+0000 7f7b98ff9700 1 -- 172.21.15.94:0/4237824778 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b9c08f460 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.421+0000 7f7baaa8f700 1 -- 172.21.15.94:0/4237824778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7ba406f280 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.423 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.421+0000 7f7b98ff9700 1 -- 172.21.15.94:0/4237824778 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7b9c092a80 con 0x7f7ba40a2ef0 2023-09-30T00:21:39.423 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:39.423 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:39.425 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.423+0000 7f7baaa8f700 10 mgrc shutdown 2023-09-30T00:21:39.425 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.423+0000 7f7baaa8f700 1 -- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b90098730 msgr2=0x7f7b9009abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:39.425 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.423+0000 7f7baaa8f700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b90098730 0x7f7b9009abe0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0003a80 tx=0x7f7ba0007ec0 comp rx=0 tx=0).stop 2023-09-30T00:21:39.425 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.423+0000 7f7baaa8f700 1 -- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ba40a2ef0 msgr2=0x7f7ba413b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:39.425 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.423+0000 7f7baaa8f700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ba40a2ef0 0x7f7ba413b760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7ba40698b0 tx=0x7f7b9c018000 comp rx=0 tx=0).stop 2023-09-30T00:21:39.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.423+0000 7f7baaa8f700 1 -- 172.21.15.94:0/4237824778 shutdown_connections 2023-09-30T00:21:39.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.423+0000 7f7baaa8f700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f7b90098730 0x7f7b9009abe0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:39.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.423+0000 7f7baaa8f700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ba40a2ef0 0x7f7ba413b760 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:39.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.423+0000 7f7baaa8f700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ba406eab0 0x7f7ba40981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:39.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.423+0000 7f7baaa8f700 1 --2- 172.21.15.94:0/4237824778 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7ba409de30 0x7f7ba4140540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:39.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.423+0000 7f7baaa8f700 1 -- 172.21.15.94:0/4237824778 >> 172.21.15.94:0/4237824778 conn(0x7f7ba40b3520 msgr2=0x7f7ba40a5210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:39.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.424+0000 7f7baaa8f700 1 -- 172.21.15.94:0/4237824778 shutdown_connections 2023-09-30T00:21:39.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:39.424+0000 7f7baaa8f700 1 -- 172.21.15.94:0/4237824778 wait complete. 2023-09-30T00:21:39.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:21:39.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:21:42.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:42.563 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.561+0000 7f4ae7599700 1 Processor -- start 2023-09-30T00:21:42.563 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.562+0000 7f4ae7599700 1 -- start start 2023-09-30T00:21:42.566 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.562+0000 7f4ae7599700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 0x7f4ae00b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:42.566 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.562+0000 7f4ae7599700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4ae00a3ff0 0x7f4ae00b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:42.566 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.562+0000 7f4ae7599700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4ae009ca70 0x7f4ae009ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:42.566 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.562+0000 7f4ae7599700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ae006ae70 con 0x7f4ae009ca70 2023-09-30T00:21:42.566 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.562+0000 7f4ae7599700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ae00b40e0 con 0x7f4ae00a3ff0 2023-09-30T00:21:42.566 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.562+0000 7f4ae7599700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae00b48c0 con 0x7f4ae006aaa0 2023-09-30T00:21:42.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.562+0000 7f4ae5b36700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4ae009ca70 0x7f4ae009ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:42.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.562+0000 7f4ae5335700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 0x7f4ae00b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:42.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.562+0000 7f4ae5335700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 0x7f4ae00b3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:53238/0 (socket says 172.21.15.94:53238) 2023-09-30T00:21:42.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.562+0000 7f4ae5b36700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4ae009ca70 0x7f4ae009ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59386/0 (socket says 172.21.15.94:59386) 2023-09-30T00:21:42.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.562+0000 7f4ae5335700 1 -- 172.21.15.94:0/139259165 learned_addr learned my addr 172.21.15.94:0/139259165 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:42.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4ae4b34700 1 --2- 172.21.15.94:0/139259165 >> v2:172.21.15.121:3300/0 conn(0x7f4ae00a3ff0 0x7f4ae00b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:42.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4ae5335700 1 -- 172.21.15.94:0/139259165 >> v2:172.21.15.121:3300/0 conn(0x7f4ae00a3ff0 msgr2=0x7f4ae00b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:42.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4ae5335700 1 --2- 172.21.15.94:0/139259165 >> v2:172.21.15.121:3300/0 conn(0x7f4ae00a3ff0 0x7f4ae00b24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:42.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4ae5335700 1 -- 172.21.15.94:0/139259165 >> v2:172.21.15.94:3300/0 conn(0x7f4ae009ca70 msgr2=0x7f4ae009ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:42.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4ae5335700 1 --2- 172.21.15.94:0/139259165 >> v2:172.21.15.94:3300/0 conn(0x7f4ae009ca70 0x7f4ae009ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:42.567 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4ae5335700 1 -- 172.21.15.94:0/139259165 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae00a45c0 con 0x7f4ae006aaa0 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4ae5b36700 1 --2- 172.21.15.94:0/139259165 >> v2:172.21.15.94:3300/0 conn(0x7f4ae009ca70 0x7f4ae009ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4ae5335700 1 --2- 172.21.15.94:0/139259165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 0x7f4ae00b3ba0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4ae006b080 tx=0x7f4ad0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83cb48b51ef9f60d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4adffff700 1 -- 172.21.15.94:0/139259165 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad0009e70 con 0x7f4ae006aaa0 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4adffff700 1 -- 172.21.15.94:0/139259165 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad000b860 con 0x7f4ae006aaa0 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4adffff700 1 -- 172.21.15.94:0/139259165 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad000bc00 con 0x7f4ae006aaa0 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4ae7599700 1 -- 172.21.15.94:0/139259165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 msgr2=0x7f4ae00b3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.563+0000 7f4ae7599700 1 --2- 172.21.15.94:0/139259165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 0x7f4ae00b3ba0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4ae006b080 tx=0x7f4ad0009250 comp rx=0 tx=0).stop 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.564+0000 7f4ae7599700 1 -- 172.21.15.94:0/139259165 shutdown_connections 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.564+0000 7f4ae7599700 1 --2- 172.21.15.94:0/139259165 >> v2:172.21.15.94:3300/0 conn(0x7f4ae009ca70 0x7f4ae009ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.564+0000 7f4ae7599700 1 --2- 172.21.15.94:0/139259165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 0x7f4ae00b3ba0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.564+0000 7f4ae7599700 1 --2- 172.21.15.94:0/139259165 >> v2:172.21.15.121:3300/0 conn(0x7f4ae00a3ff0 0x7f4ae00b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.564+0000 7f4ae7599700 1 -- 172.21.15.94:0/139259165 >> 172.21.15.94:0/139259165 conn(0x7f4ae0068730 msgr2=0x7f4ae0068b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:42.568 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.564+0000 7f4ae7599700 1 -- 172.21.15.94:0/139259165 shutdown_connections 2023-09-30T00:21:42.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.564+0000 7f4ae7599700 1 -- 172.21.15.94:0/139259165 wait complete. 2023-09-30T00:21:42.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.565+0000 7f4ae7599700 1 Processor -- start 2023-09-30T00:21:42.569 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.565+0000 7f4ae7599700 1 -- start start 2023-09-30T00:21:42.571 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.565+0000 7f4ae7599700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 0x7f4ae0067870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:42.571 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.565+0000 7f4ae7599700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ae009ca70 0x7f4ae009dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:42.572 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.565+0000 7f4ae7599700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ae00a3ff0 0x7f4ae009eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:42.572 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.565+0000 7f4ae7599700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae0146c70 con 0x7f4ae00a3ff0 2023-09-30T00:21:42.572 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.565+0000 7f4ae7599700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae009c650 con 0x7f4ae009ca70 2023-09-30T00:21:42.572 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.565+0000 7f4ae7599700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae01450c0 con 0x7f4ae006aaa0 2023-09-30T00:21:42.572 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.565+0000 7f4ae5335700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 0x7f4ae0067870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:42.572 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.565+0000 7f4ae5b36700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ae00a3ff0 0x7f4ae009eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:42.572 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.565+0000 7f4ae5335700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 0x7f4ae0067870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:53250/0 (socket says 172.21.15.94:53250) 2023-09-30T00:21:42.572 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.565+0000 7f4ae5335700 1 -- 172.21.15.94:0/912190981 learned_addr learned my addr 172.21.15.94:0/912190981 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:42.572 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4ae4b34700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ae009ca70 0x7f4ae009dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:42.572 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4ae5335700 1 -- 172.21.15.94:0/912190981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ae009ca70 msgr2=0x7f4ae009dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:42.572 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4ae5335700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ae009ca70 0x7f4ae009dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:42.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4ae5335700 1 -- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ae00a3ff0 msgr2=0x7f4ae009eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:42.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4ae5335700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ae00a3ff0 0x7f4ae009eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:42.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4ae5335700 1 -- 172.21.15.94:0/912190981 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad8009a20 con 0x7f4ae006aaa0 2023-09-30T00:21:42.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4ae5b36700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ae00a3ff0 0x7f4ae009eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:42.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4ae5335700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 0x7f4ae0067870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4ae00b4a00 tx=0x7f4ad0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:42.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4add7fa700 1 -- 172.21.15.94:0/912190981 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad000bbe0 con 0x7f4ae006aaa0 2023-09-30T00:21:42.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4add7fa700 1 -- 172.21.15.94:0/912190981 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad0011420 con 0x7f4ae006aaa0 2023-09-30T00:21:42.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4ae7599700 1 -- 172.21.15.94:0/912190981 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad00094f0 con 0x7f4ae006aaa0 2023-09-30T00:21:42.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4add7fa700 1 -- 172.21.15.94:0/912190981 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad00117c0 con 0x7f4ae006aaa0 2023-09-30T00:21:42.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.566+0000 7f4ae7599700 1 -- 172.21.15.94:0/912190981 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae0068140 con 0x7f4ae006aaa0 2023-09-30T00:21:42.573 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.567+0000 7f4ae7599700 1 -- 172.21.15.94:0/912190981 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae009e800 con 0x7f4ae006aaa0 2023-09-30T00:21:42.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.568+0000 7f4add7fa700 1 -- 172.21.15.94:0/912190981 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f4ad0011bb0 con 0x7f4ae006aaa0 2023-09-30T00:21:42.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.568+0000 7f4add7fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:42.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.568+0000 7f4add7fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:42.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.568+0000 7f4add7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:42.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.568+0000 7f4add7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:42.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.568+0000 7f4add7fa700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4acc098780 0x7f4acc09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:42.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.568+0000 7f4ae4b34700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4acc098780 0x7f4acc09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:42.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.568+0000 7f4add7fa700 1 -- 172.21.15.94:0/912190981 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f4ad00c1c60 con 0x7f4ae006aaa0 2023-09-30T00:21:42.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.569+0000 7f4ae4b34700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4acc098780 0x7f4acc09ac30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4ae009ca00 tx=0x7f4ad4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:42.575 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.572+0000 7f4add7fa700 1 -- 172.21.15.94:0/912190981 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ae009e800 con 0x7f4ae006aaa0 2023-09-30T00:21:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.794+0000 7f4ae7599700 1 -- 172.21.15.94:0/912190981 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4ae009e800 con 0x7f4ae006aaa0 2023-09-30T00:21:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.795+0000 7f4add7fa700 1 -- 172.21.15.94:0/912190981 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4ae009e800 con 0x7f4ae006aaa0 2023-09-30T00:21:42.797 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:42.797 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 10 mgrc shutdown 2023-09-30T00:21:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 1 -- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4acc098780 msgr2=0x7f4acc09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4acc098780 0x7f4acc09ac30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4ae009ca00 tx=0x7f4ad4008040 comp rx=0 tx=0).stop 2023-09-30T00:21:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 1 -- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 msgr2=0x7f4ae0067870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 0x7f4ae0067870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4ae00b4a00 tx=0x7f4ad0017000 comp rx=0 tx=0).stop 2023-09-30T00:21:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 1 -- 172.21.15.94:0/912190981 shutdown_connections 2023-09-30T00:21:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4acc098780 0x7f4acc09ac30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ae006aaa0 0x7f4ae0067870 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ae00a3ff0 0x7f4ae009eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:42.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 1 --2- 172.21.15.94:0/912190981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ae009ca70 0x7f4ae009dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:42.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 1 -- 172.21.15.94:0/912190981 >> 172.21.15.94:0/912190981 conn(0x7f4ae0068730 msgr2=0x7f4ae013ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:42.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 1 -- 172.21.15.94:0/912190981 shutdown_connections 2023-09-30T00:21:42.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:42.797+0000 7f4ae7599700 1 -- 172.21.15.94:0/912190981 wait complete. 2023-09-30T00:21:45.810 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:45.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.933+0000 7fdd3a9d3700 1 Processor -- start 2023-09-30T00:21:45.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.933+0000 7fdd3a9d3700 1 -- start start 2023-09-30T00:21:45.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.933+0000 7fdd3a9d3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd340b7160 0x7fdd340a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:45.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.933+0000 7fdd3a9d3700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd340a1310 con 0x7fdd34070c50 2023-09-30T00:21:45.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd3a9d3700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd34071020 con 0x7fdd3413fd70 2023-09-30T00:21:45.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd3a9d3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd340b76a0 con 0x7fdd340b7160 2023-09-30T00:21:45.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd33fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdd34070c50 0x7fdd340a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:55918/0 (socket says 172.21.15.94:55918) 2023-09-30T00:21:45.938 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd33fff700 1 -- 172.21.15.94:0/3607788621 learned_addr learned my addr 172.21.15.94:0/3607788621 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:21:45.938 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd337fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd340b7160 0x7fdd340a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:45.938 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd32ffd700 1 -- 172.21.15.94:0/3607788621 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962882158 0 0) 0x7fdd24001110 con 0x7fdd34070c50 2023-09-30T00:21:45.938 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd32ffd700 1 -- 172.21.15.94:0/3607788621 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd1c003570 con 0x7fdd34070c50 2023-09-30T00:21:45.938 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd337fe700 1 -- 172.21.15.94:0/3607788621 >> v1:172.21.15.121:6789/0 conn(0x7fdd3413fd70 legacy=0x7fdd3414b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:45.938 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd337fe700 1 -- 172.21.15.94:0/3607788621 >> v1:172.21.15.94:6789/0 conn(0x7fdd34070c50 legacy=0x7fdd340a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:45.938 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd337fe700 1 -- 172.21.15.94:0/3607788621 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd340b7940 con 0x7fdd340b7160 2023-09-30T00:21:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd337fe700 1 --2- 172.21.15.94:0/3607788621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd340b7160 0x7fdd340a0dd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009910 tx=0x7fdd280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a914adaf74ddbddb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd32ffd700 1 -- 172.21.15.94:0/3607788621 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2800b710 con 0x7fdd340b7160 2023-09-30T00:21:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd32ffd700 1 -- 172.21.15.94:0/3607788621 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2800b870 con 0x7fdd340b7160 2023-09-30T00:21:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.934+0000 7fdd32ffd700 1 -- 172.21.15.94:0/3607788621 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2800bc30 con 0x7fdd340b7160 2023-09-30T00:21:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.935+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/3607788621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd340b7160 msgr2=0x7fdd340a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.935+0000 7fdd3a9d3700 1 --2- 172.21.15.94:0/3607788621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd340b7160 0x7fdd340a0dd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009910 tx=0x7fdd280091a0 comp rx=0 tx=0).stop 2023-09-30T00:21:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.935+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/3607788621 shutdown_connections 2023-09-30T00:21:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.935+0000 7fdd3a9d3700 1 --2- 172.21.15.94:0/3607788621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd340b7160 0x7fdd340a0dd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:45.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.935+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/3607788621 >> 172.21.15.94:0/3607788621 conn(0x7fdd3406a090 msgr2=0x7fdd340702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.935+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/3607788621 shutdown_connections 2023-09-30T00:21:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.935+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/3607788621 wait complete. 2023-09-30T00:21:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.936+0000 7fdd3a9d3700 1 Processor -- start 2023-09-30T00:21:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.936+0000 7fdd3a9d3700 1 -- start start 2023-09-30T00:21:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd3a9d3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34070c50 0x7fdd340a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd3a9d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd340b7160 0x7fdd34143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd3a9d3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd3413fd70 0x7fdd340ab5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd3a9d3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd340a6df0 con 0x7fdd34070c50 2023-09-30T00:21:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd3a9d3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd340a6c30 con 0x7fdd340b7160 2023-09-30T00:21:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd3a9d3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd340a4010 con 0x7fdd3413fd70 2023-09-30T00:21:45.943 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd33fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34070c50 0x7fdd340a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd38f70700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd3413fd70 0x7fdd340ab5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd33fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34070c50 0x7fdd340a66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41904/0 (socket says 172.21.15.94:41904) 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd38f70700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd3413fd70 0x7fdd340ab5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44750/0 (socket says 172.21.15.94:44750) 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd33fff700 1 -- 172.21.15.94:0/2250991907 learned_addr learned my addr 172.21.15.94:0/2250991907 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd337fe700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd340b7160 0x7fdd34143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd33fff700 1 -- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd3413fd70 msgr2=0x7fdd340ab5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd33fff700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd3413fd70 0x7fdd340ab5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd33fff700 1 -- 172.21.15.94:0/2250991907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd340b7160 msgr2=0x7fdd34143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd33fff700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd340b7160 0x7fdd34143bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd33fff700 1 -- 172.21.15.94:0/2250991907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd28009420 con 0x7fdd34070c50 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd38f70700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd3413fd70 0x7fdd340ab5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd33fff700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34070c50 0x7fdd340a66f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400c7c0 tx=0x7fdd24007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:45.944 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd1bfff700 1 -- 172.21.15.94:0/2250991907 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24005b30 con 0x7fdd34070c50 2023-09-30T00:21:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd1bfff700 1 -- 172.21.15.94:0/2250991907 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd24005c90 con 0x7fdd34070c50 2023-09-30T00:21:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/2250991907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd34144130 con 0x7fdd34070c50 2023-09-30T00:21:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd1bfff700 1 -- 172.21.15.94:0/2250991907 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24007320 con 0x7fdd34070c50 2023-09-30T00:21:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.937+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/2250991907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd34144490 con 0x7fdd34070c50 2023-09-30T00:21:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.938+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/2250991907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd340ab290 con 0x7fdd34070c50 2023-09-30T00:21:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.939+0000 7fdd1bfff700 1 -- 172.21.15.94:0/2250991907 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fdd2401b3b0 con 0x7fdd34070c50 2023-09-30T00:21:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.939+0000 7fdd1bfff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.940+0000 7fdd1bfff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.940+0000 7fdd1bfff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.940+0000 7fdd1bfff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.940+0000 7fdd1bfff700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdd20098780 0x7fdd2009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:45.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.940+0000 7fdd337fe700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdd20098780 0x7fdd2009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.940+0000 7fdd1bfff700 1 -- 172.21.15.94:0/2250991907 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fdd240bf7c0 con 0x7fdd34070c50 2023-09-30T00:21:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.940+0000 7fdd337fe700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdd20098780 0x7fdd2009ac30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdd280091e0 tx=0x7fdd28017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:45.944+0000 7fdd1bfff700 1 -- 172.21.15.94:0/2250991907 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdd2408cdf0 con 0x7fdd34070c50 2023-09-30T00:21:46.165 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.164+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/2250991907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdd340a1310 con 0x7fdd34070c50 2023-09-30T00:21:46.165 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.164+0000 7fdd1bfff700 1 -- 172.21.15.94:0/2250991907 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdd24017100 con 0x7fdd34070c50 2023-09-30T00:21:46.166 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:46.166 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:46.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 10 mgrc shutdown 2023-09-30T00:21:46.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdd20098780 msgr2=0x7fdd2009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:46.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdd20098780 0x7fdd2009ac30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdd280091e0 tx=0x7fdd28017000 comp rx=0 tx=0).stop 2023-09-30T00:21:46.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34070c50 msgr2=0x7fdd340a66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:46.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34070c50 0x7fdd340a66f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400c7c0 tx=0x7fdd24007c70 comp rx=0 tx=0).stop 2023-09-30T00:21:46.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/2250991907 shutdown_connections 2023-09-30T00:21:46.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fdd20098780 0x7fdd2009ac30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:46.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdd3413fd70 0x7fdd340ab5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:46.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34070c50 0x7fdd340a66f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:46.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 1 --2- 172.21.15.94:0/2250991907 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd340b7160 0x7fdd34143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:46.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/2250991907 >> 172.21.15.94:0/2250991907 conn(0x7fdd3406a090 msgr2=0x7fdd340a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:46.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/2250991907 shutdown_connections 2023-09-30T00:21:46.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:46.167+0000 7fdd3a9d3700 1 -- 172.21.15.94:0/2250991907 wait complete. 2023-09-30T00:21:49.180 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.303+0000 7fcbe83e4700 1 Processor -- start 2023-09-30T00:21:49.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.303+0000 7fcbe83e4700 1 -- start start 2023-09-30T00:21:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.304+0000 7fcbe83e4700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcbe0070c50 0x7fcbe00a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:49.309 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.304+0000 7fcbe83e4700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcbe013fd70 0x7fcbe0140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.304+0000 7fcbe83e4700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe00a1310 con 0x7fcbe00b7160 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.304+0000 7fcbe83e4700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe0071020 con 0x7fcbe013fd70 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.304+0000 7fcbe83e4700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe00b76a0 con 0x7fcbe0070c50 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.304+0000 7fcbe6981700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcbe013fd70 0x7fcbe0140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.304+0000 7fcbe597f700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcbe00b7160 0x7fcbe00a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:55922/0 (socket says 172.21.15.94:55922) 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.304+0000 7fcbe597f700 1 -- 172.21.15.94:0/1456560588 learned_addr learned my addr 172.21.15.94:0/1456560588 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.305+0000 7fcbe6180700 1 --2- 172.21.15.94:0/1456560588 >> v2:172.21.15.121:3300/0 conn(0x7fcbe0070c50 0x7fcbe00a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.305+0000 7fcbe6981700 1 -- 172.21.15.94:0/1456560588 >> v2:172.21.15.121:3300/0 conn(0x7fcbe0070c50 msgr2=0x7fcbe00a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.305+0000 7fcbe6981700 1 --2- 172.21.15.94:0/1456560588 >> v2:172.21.15.121:3300/0 conn(0x7fcbe0070c50 0x7fcbe00a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.305+0000 7fcbe6981700 1 -- 172.21.15.94:0/1456560588 >> v1:172.21.15.94:6789/0 conn(0x7fcbe00b7160 legacy=0x7fcbe00a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.305+0000 7fcbe6981700 1 -- 172.21.15.94:0/1456560588 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe00b7940 con 0x7fcbe013fd70 2023-09-30T00:21:49.310 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.305+0000 7fcbe517e700 1 -- 172.21.15.94:0/1456560588 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215422211 0 0) 0x7fcbd0001110 con 0x7fcbe00b7160 2023-09-30T00:21:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.305+0000 7fcbe6981700 1 --2- 172.21.15.94:0/1456560588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbe013fd70 0x7fcbe0140180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00a9760 tx=0x7fcbd80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1577f67517cb747e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.305+0000 7fcbe517e700 1 -- 172.21.15.94:0/1456560588 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd800b740 con 0x7fcbe013fd70 2023-09-30T00:21:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.305+0000 7fcbe517e700 1 -- 172.21.15.94:0/1456560588 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd800b8a0 con 0x7fcbe013fd70 2023-09-30T00:21:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.305+0000 7fcbe517e700 1 -- 172.21.15.94:0/1456560588 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd800bc40 con 0x7fcbe013fd70 2023-09-30T00:21:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.306+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1456560588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbe013fd70 msgr2=0x7fcbe0140180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.306+0000 7fcbe83e4700 1 --2- 172.21.15.94:0/1456560588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbe013fd70 0x7fcbe0140180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00a9760 tx=0x7fcbd80091a0 comp rx=0 tx=0).stop 2023-09-30T00:21:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.306+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1456560588 shutdown_connections 2023-09-30T00:21:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.306+0000 7fcbe83e4700 1 --2- 172.21.15.94:0/1456560588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbe013fd70 0x7fcbe0140180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.306+0000 7fcbe83e4700 1 --2- 172.21.15.94:0/1456560588 >> v2:172.21.15.121:3300/0 conn(0x7fcbe0070c50 0x7fcbe00a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.306+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1456560588 >> 172.21.15.94:0/1456560588 conn(0x7fcbe006a090 msgr2=0x7fcbe00702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:49.311 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.306+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1456560588 shutdown_connections 2023-09-30T00:21:49.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.306+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1456560588 wait complete. 2023-09-30T00:21:49.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe83e4700 1 Processor -- start 2023-09-30T00:21:49.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe83e4700 1 -- start start 2023-09-30T00:21:49.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe83e4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcbe0070c50 0x7fcbe0142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:49.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe83e4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbe00b7160 0x7fcbe0143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:49.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe83e4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbe013fd70 0x7fcbe0143670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:49.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe83e4700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe00a6040 con 0x7fcbe00b7160 2023-09-30T00:21:49.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe83e4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe00a6320 con 0x7fcbe013fd70 2023-09-30T00:21:49.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe83e4700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe00a3f60 con 0x7fcbe0070c50 2023-09-30T00:21:49.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe6180700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcbe0070c50 0x7fcbe0142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:49.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe597f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbe00b7160 0x7fcbe0143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:49.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe6180700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcbe0070c50 0x7fcbe0142bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44754/0 (socket says 172.21.15.94:44754) 2023-09-30T00:21:49.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe6180700 1 -- 172.21.15.94:0/1844033724 learned_addr learned my addr 172.21.15.94:0/1844033724 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:49.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.307+0000 7fcbe6981700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbe013fd70 0x7fcbe0143670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:49.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.308+0000 7fcbe6180700 1 -- 172.21.15.94:0/1844033724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbe013fd70 msgr2=0x7fcbe0143670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:49.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.308+0000 7fcbe6180700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbe013fd70 0x7fcbe0143670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:49.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.308+0000 7fcbe6180700 1 -- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbe00b7160 msgr2=0x7fcbe0143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:49.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.308+0000 7fcbe6180700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbe00b7160 0x7fcbe0143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:49.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.308+0000 7fcbe6180700 1 -- 172.21.15.94:0/1844033724 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd8009420 con 0x7fcbe0070c50 2023-09-30T00:21:49.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.308+0000 7fcbe597f700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbe00b7160 0x7fcbe0143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:49.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.308+0000 7fcbe6180700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcbe0070c50 0x7fcbe0142bf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00e190 tx=0x7fcbdc009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:49.313 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.308+0000 7fcbd67fc700 1 -- 172.21.15.94:0/1844033724 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc00ecd0 con 0x7fcbe0070c50 2023-09-30T00:21:49.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.308+0000 7fcbd67fc700 1 -- 172.21.15.94:0/1844033724 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbdc00ee30 con 0x7fcbe0070c50 2023-09-30T00:21:49.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.308+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1844033724 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe00a41e0 con 0x7fcbe0070c50 2023-09-30T00:21:49.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.308+0000 7fcbd67fc700 1 -- 172.21.15.94:0/1844033724 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc014670 con 0x7fcbe0070c50 2023-09-30T00:21:49.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.308+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1844033724 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe009bd70 con 0x7fcbe0070c50 2023-09-30T00:21:49.314 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.309+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1844033724 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe009ff70 con 0x7fcbe0070c50 2023-09-30T00:21:49.315 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.310+0000 7fcbd67fc700 1 -- 172.21.15.94:0/1844033724 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fcbdc014a20 con 0x7fcbe0070c50 2023-09-30T00:21:49.315 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.310+0000 7fcbd67fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:49.315 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.310+0000 7fcbd67fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:49.315 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.310+0000 7fcbd67fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:49.315 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.310+0000 7fcbd67fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:49.315 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.310+0000 7fcbd67fc700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcbc4098780 0x7fcbc409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:49.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.310+0000 7fcbe597f700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcbc4098780 0x7fcbc409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:49.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.310+0000 7fcbd67fc700 1 -- 172.21.15.94:0/1844033724 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fcbdc0c16d0 con 0x7fcbe0070c50 2023-09-30T00:21:49.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.311+0000 7fcbe597f700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcbc4098780 0x7fcbc409ac30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00a05d0 tx=0x7fcbd0007b90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:49.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.313+0000 7fcbd67fc700 1 -- 172.21.15.94:0/1844033724 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcbdc08ed10 con 0x7fcbe0070c50 2023-09-30T00:21:49.538 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.538+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1844033724 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcbe009fd90 con 0x7fcbe0070c50 2023-09-30T00:21:49.539 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.538+0000 7fcbd67fc700 1 -- 172.21.15.94:0/1844033724 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcbdc092330 con 0x7fcbe0070c50 2023-09-30T00:21:49.539 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:49.539 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:49.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 10 mgrc shutdown 2023-09-30T00:21:49.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcbc4098780 msgr2=0x7fcbc409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:49.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcbc4098780 0x7fcbc409ac30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00a05d0 tx=0x7fcbd0007b90 comp rx=0 tx=0).stop 2023-09-30T00:21:49.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcbe0070c50 msgr2=0x7fcbe0142bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:49.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcbe0070c50 0x7fcbe0142bf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00e190 tx=0x7fcbdc009920 comp rx=0 tx=0).stop 2023-09-30T00:21:49.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1844033724 shutdown_connections 2023-09-30T00:21:49.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcbc4098780 0x7fcbc409ac30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:49.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcbe0070c50 0x7fcbe0142bf0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:49.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbe013fd70 0x7fcbe0143670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:49.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 1 --2- 172.21.15.94:0/1844033724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbe00b7160 0x7fcbe0143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:49.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1844033724 >> 172.21.15.94:0/1844033724 conn(0x7fcbe006a090 msgr2=0x7fcbe0140f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:49.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1844033724 shutdown_connections 2023-09-30T00:21:49.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:49.540+0000 7fcbe83e4700 1 -- 172.21.15.94:0/1844033724 wait complete. 2023-09-30T00:21:52.553 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:52.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a0ea09700 1 Processor -- start 2023-09-30T00:21:52.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a0ea09700 1 -- start start 2023-09-30T00:21:52.678 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a0ea09700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3a080b7160 0x7f3a080a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:52.678 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a0ea09700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a0813fd70 0x7f3a081401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:52.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a0ea09700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a080a1310 con 0x7f3a08070c50 2023-09-30T00:21:52.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a0ea09700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a08071020 con 0x7f3a080b7160 2023-09-30T00:21:52.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a0ea09700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a080b76a0 con 0x7f3a0813fd70 2023-09-30T00:21:52.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a077fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3a080b7160 0x7f3a080a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:52.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a0cfa6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a0813fd70 0x7f3a081401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:52.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a077fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3a080b7160 0x7f3a080a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41936/0 (socket says 172.21.15.94:41936) 2023-09-30T00:21:52.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a0cfa6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a0813fd70 0x7f3a081401d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:44770/0 (socket says 172.21.15.94:44770) 2023-09-30T00:21:52.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a077fe700 1 -- 172.21.15.94:0/982073401 learned_addr learned my addr 172.21.15.94:0/982073401 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:52.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.676+0000 7f3a07fff700 1 --1- 172.21.15.94:0/982073401 >> v1:172.21.15.94:6789/0 conn(0x7f3a08070c50 0x7f3a080a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:55926/0 (socket says 172.21.15.94:55926) 2023-09-30T00:21:52.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.677+0000 7f3a06ffd700 1 -- 172.21.15.94:0/982073401 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562587304 0 0) 0x7f39f8001110 con 0x7f3a08070c50 2023-09-30T00:21:52.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.677+0000 7f3a0cfa6700 1 -- 172.21.15.94:0/982073401 >> v2:172.21.15.94:3300/0 conn(0x7f3a080b7160 msgr2=0x7f3a080a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:52.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.677+0000 7f3a0cfa6700 1 --2- 172.21.15.94:0/982073401 >> v2:172.21.15.94:3300/0 conn(0x7f3a080b7160 0x7f3a080a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.677+0000 7f3a0cfa6700 1 -- 172.21.15.94:0/982073401 >> v1:172.21.15.94:6789/0 conn(0x7f3a08070c50 legacy=0x7f3a080a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.677+0000 7f3a0cfa6700 1 -- 172.21.15.94:0/982073401 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a080b7940 con 0x7f3a0813fd70 2023-09-30T00:21:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.677+0000 7f3a077fe700 1 --2- 172.21.15.94:0/982073401 >> v2:172.21.15.94:3300/0 conn(0x7f3a080b7160 0x7f3a080a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.677+0000 7f3a0cfa6700 1 --2- 172.21.15.94:0/982073401 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a0813fd70 0x7f3a081401d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009e60 tx=0x7f3a00009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bfd0fb356312bd0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.677+0000 7f3a06ffd700 1 -- 172.21.15.94:0/982073401 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0000e070 con 0x7f3a0813fd70 2023-09-30T00:21:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.677+0000 7f3a06ffd700 1 -- 172.21.15.94:0/982073401 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0000bbf0 con 0x7f3a0813fd70 2023-09-30T00:21:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.677+0000 7f3a06ffd700 1 -- 172.21.15.94:0/982073401 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00012430 con 0x7f3a0813fd70 2023-09-30T00:21:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.678+0000 7f3a0ea09700 1 -- 172.21.15.94:0/982073401 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a0813fd70 msgr2=0x7f3a081401d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.678+0000 7f3a0ea09700 1 --2- 172.21.15.94:0/982073401 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a0813fd70 0x7f3a081401d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009e60 tx=0x7f3a00009780 comp rx=0 tx=0).stop 2023-09-30T00:21:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.678+0000 7f3a0ea09700 1 -- 172.21.15.94:0/982073401 shutdown_connections 2023-09-30T00:21:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.678+0000 7f3a0ea09700 1 --2- 172.21.15.94:0/982073401 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a0813fd70 0x7f3a081401d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.678+0000 7f3a0ea09700 1 --2- 172.21.15.94:0/982073401 >> v2:172.21.15.94:3300/0 conn(0x7f3a080b7160 0x7f3a080a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:52.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.678+0000 7f3a0ea09700 1 -- 172.21.15.94:0/982073401 >> 172.21.15.94:0/982073401 conn(0x7f3a0806a090 msgr2=0x7f3a080702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:52.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.678+0000 7f3a0ea09700 1 -- 172.21.15.94:0/982073401 shutdown_connections 2023-09-30T00:21:52.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.678+0000 7f3a0ea09700 1 -- 172.21.15.94:0/982073401 wait complete. 2023-09-30T00:21:52.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.678+0000 7f3a0ea09700 1 Processor -- start 2023-09-30T00:21:52.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.679+0000 7f3a0ea09700 1 -- start start 2023-09-30T00:21:52.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.679+0000 7f3a0ea09700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a08070c50 0x7f3a0813bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:52.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.679+0000 7f3a0ea09700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a080b7160 0x7f3a08137b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:52.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.679+0000 7f3a0ea09700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a0813fd70 0x7f3a08138080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:52.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.679+0000 7f3a0ea09700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0813c500 con 0x7f3a0813fd70 2023-09-30T00:21:52.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.679+0000 7f3a0ea09700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a081548d0 con 0x7f3a080b7160 2023-09-30T00:21:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.679+0000 7f3a0ea09700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a080a6980 con 0x7f3a08070c50 2023-09-30T00:21:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.679+0000 7f3a07fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a08070c50 0x7f3a0813bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.679+0000 7f3a0cfa6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a0813fd70 0x7f3a08138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.679+0000 7f3a0cfa6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a0813fd70 0x7f3a08138080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41940/0 (socket says 172.21.15.94:41940) 2023-09-30T00:21:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.679+0000 7f3a0cfa6700 1 -- 172.21.15.94:0/3708724360 learned_addr learned my addr 172.21.15.94:0/3708724360 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f3a077fe700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a080b7160 0x7f3a08137b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f3a07fff700 1 -- 172.21.15.94:0/3708724360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a080b7160 msgr2=0x7f3a08137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f3a07fff700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a080b7160 0x7f3a08137b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f3a07fff700 1 -- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a0813fd70 msgr2=0x7f3a08138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f3a07fff700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a0813fd70 0x7f3a08138080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f3a07fff700 1 -- 172.21.15.94:0/3708724360 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39fc0094f0 con 0x7f3a08070c50 2023-09-30T00:21:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f3a0cfa6700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a0813fd70 0x7f3a08138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f3a07fff700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a08070c50 0x7f3a0813bef0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f39f800c810 tx=0x7f39f8007b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f39effff700 1 -- 172.21.15.94:0/3708724360 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f8005a70 con 0x7f3a08070c50 2023-09-30T00:21:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f39effff700 1 -- 172.21.15.94:0/3708724360 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f8005bd0 con 0x7f3a08070c50 2023-09-30T00:21:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f3a0ea09700 1 -- 172.21.15.94:0/3708724360 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a00009a20 con 0x7f3a08070c50 2023-09-30T00:21:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f39effff700 1 -- 172.21.15.94:0/3708724360 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f80070d0 con 0x7f3a08070c50 2023-09-30T00:21:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.680+0000 7f3a0ea09700 1 -- 172.21.15.94:0/3708724360 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a080a57f0 con 0x7f3a08070c50 2023-09-30T00:21:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.681+0000 7f3a0ea09700 1 -- 172.21.15.94:0/3708724360 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a08071630 con 0x7f3a08070c50 2023-09-30T00:21:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.682+0000 7f39effff700 1 -- 172.21.15.94:0/3708724360 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f39f80074c0 con 0x7f3a08070c50 2023-09-30T00:21:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.682+0000 7f39effff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.682+0000 7f39effff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.682+0000 7f39effff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:52.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.682+0000 7f39effff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:52.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.682+0000 7f39effff700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f39f4098700 0x7f39f409abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:52.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.683+0000 7f3a077fe700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f39f4098700 0x7f39f409abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:52.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.683+0000 7f39effff700 1 -- 172.21.15.94:0/3708724360 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f39f80bf8d0 con 0x7f3a08070c50 2023-09-30T00:21:52.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.683+0000 7f3a077fe700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f39f4098700 0x7f39f409abb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0094c0 tx=0x7f39fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:52.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.687+0000 7f39effff700 1 -- 172.21.15.94:0/3708724360 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f39f808ce70 con 0x7f3a08070c50 2023-09-30T00:21:52.912 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.911+0000 7f3a0ea09700 1 -- 172.21.15.94:0/3708724360 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3a080a1310 con 0x7f3a08070c50 2023-09-30T00:21:52.912 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.911+0000 7f39effff700 1 -- 172.21.15.94:0/3708724360 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3a080a1310 con 0x7f3a08070c50 2023-09-30T00:21:52.912 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:52.912 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:52.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.913+0000 7f3a0ea09700 10 mgrc shutdown 2023-09-30T00:21:52.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.913+0000 7f3a0ea09700 1 -- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f39f4098700 msgr2=0x7f39f409abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:52.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.913+0000 7f3a0ea09700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f39f4098700 0x7f39f409abb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0094c0 tx=0x7f39fc008040 comp rx=0 tx=0).stop 2023-09-30T00:21:52.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.913+0000 7f3a0ea09700 1 -- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a08070c50 msgr2=0x7f3a0813bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:52.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.913+0000 7f3a0ea09700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a08070c50 0x7f3a0813bef0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f39f800c810 tx=0x7f39f8007b90 comp rx=0 tx=0).stop 2023-09-30T00:21:52.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.913+0000 7f3a0ea09700 1 -- 172.21.15.94:0/3708724360 shutdown_connections 2023-09-30T00:21:52.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.913+0000 7f3a0ea09700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f39f4098700 0x7f39f409abb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:52.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.913+0000 7f3a0ea09700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a08070c50 0x7f3a0813bef0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:52.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.913+0000 7f3a0ea09700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a0813fd70 0x7f3a08138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:52.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.913+0000 7f3a0ea09700 1 --2- 172.21.15.94:0/3708724360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a080b7160 0x7f3a08137b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:52.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.914+0000 7f3a0ea09700 1 -- 172.21.15.94:0/3708724360 >> 172.21.15.94:0/3708724360 conn(0x7f3a0806a090 msgr2=0x7f3a080685c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:52.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.914+0000 7f3a0ea09700 1 -- 172.21.15.94:0/3708724360 shutdown_connections 2023-09-30T00:21:52.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:52.914+0000 7f3a0ea09700 1 -- 172.21.15.94:0/3708724360 wait complete. 2023-09-30T00:21:55.927 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:56.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.051+0000 7fb5b986b700 1 Processor -- start 2023-09-30T00:21:56.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.052+0000 7fb5b986b700 1 -- start start 2023-09-30T00:21:56.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.052+0000 7fb5b986b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb5b4070c50 0x7fb5b40a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:56.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.052+0000 7fb5b986b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb5b413fd70 0x7fb5b4140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:56.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.052+0000 7fb5b986b700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b40a1310 con 0x7fb5b40b7160 2023-09-30T00:21:56.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.052+0000 7fb5b986b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b4071020 con 0x7fb5b4070c50 2023-09-30T00:21:56.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.052+0000 7fb5b986b700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b40b76a0 con 0x7fb5b413fd70 2023-09-30T00:21:56.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.052+0000 7fb5b2ffd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb5b4070c50 0x7fb5b40a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:56.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.052+0000 7fb5b2ffd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb5b4070c50 0x7fb5b40a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47386/0 (socket says 172.21.15.94:47386) 2023-09-30T00:21:56.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.052+0000 7fb5b2ffd700 1 -- 172.21.15.94:0/2218461659 learned_addr learned my addr 172.21.15.94:0/2218461659 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:56.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.052+0000 7fb5b37fe700 1 --2- 172.21.15.94:0/2218461659 >> v2:172.21.15.121:3300/0 conn(0x7fb5b413fd70 0x7fb5b4140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:56.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b2ffd700 1 -- 172.21.15.94:0/2218461659 >> v2:172.21.15.121:3300/0 conn(0x7fb5b413fd70 msgr2=0x7fb5b4140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:56.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b2ffd700 1 --2- 172.21.15.94:0/2218461659 >> v2:172.21.15.121:3300/0 conn(0x7fb5b413fd70 0x7fb5b4140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:56.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b2ffd700 1 -- 172.21.15.94:0/2218461659 >> v1:172.21.15.121:6789/0 conn(0x7fb5b40b7160 legacy=0x7fb5b40a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:56.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b2ffd700 1 -- 172.21.15.94:0/2218461659 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b40b7940 con 0x7fb5b4070c50 2023-09-30T00:21:56.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b2ffd700 1 --2- 172.21.15.94:0/2218461659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5b4070c50 0x7fb5b40a0dd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40a31a0 tx=0x7fb5a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d585205f580fa59d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:56.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b27fc700 1 -- 172.21.15.94:0/2218461659 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a400b8b0 con 0x7fb5b4070c50 2023-09-30T00:21:56.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b27fc700 1 -- 172.21.15.94:0/2218461659 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a400ba10 con 0x7fb5b4070c50 2023-09-30T00:21:56.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b27fc700 1 -- 172.21.15.94:0/2218461659 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a400bdb0 con 0x7fb5b4070c50 2023-09-30T00:21:56.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b986b700 1 -- 172.21.15.94:0/2218461659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5b4070c50 msgr2=0x7fb5b40a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:56.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b986b700 1 --2- 172.21.15.94:0/2218461659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5b4070c50 0x7fb5b40a0dd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40a31a0 tx=0x7fb5a4009250 comp rx=0 tx=0).stop 2023-09-30T00:21:56.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b986b700 1 -- 172.21.15.94:0/2218461659 shutdown_connections 2023-09-30T00:21:56.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b986b700 1 --2- 172.21.15.94:0/2218461659 >> v2:172.21.15.121:3300/0 conn(0x7fb5b413fd70 0x7fb5b4140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:56.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b986b700 1 --2- 172.21.15.94:0/2218461659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5b4070c50 0x7fb5b40a0dd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:56.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b986b700 1 -- 172.21.15.94:0/2218461659 >> 172.21.15.94:0/2218461659 conn(0x7fb5b406a090 msgr2=0x7fb5b40702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:56.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b986b700 1 -- 172.21.15.94:0/2218461659 shutdown_connections 2023-09-30T00:21:56.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.053+0000 7fb5b986b700 1 -- 172.21.15.94:0/2218461659 wait complete. 2023-09-30T00:21:56.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5b986b700 1 Processor -- start 2023-09-30T00:21:56.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5b986b700 1 -- start start 2023-09-30T00:21:56.058 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5b986b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb5b4070c50 0x7fb5b4139e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:56.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5b986b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5b40b7160 0x7fb5b40a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:56.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5b986b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5b413fd70 0x7fb5b4142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:56.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5b986b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b4144710 con 0x7fb5b40b7160 2023-09-30T00:21:56.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5b986b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b40a0290 con 0x7fb5b413fd70 2023-09-30T00:21:56.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5b986b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b40a4fe0 con 0x7fb5b4070c50 2023-09-30T00:21:56.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5aa7fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5b40b7160 0x7fb5b40a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:56.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5b2ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb5b4070c50 0x7fb5b4139e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:56.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5aa7fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5b40b7160 0x7fb5b40a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47398/0 (socket says 172.21.15.94:47398) 2023-09-30T00:21:56.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5aa7fc700 1 -- 172.21.15.94:0/771391100 learned_addr learned my addr 172.21.15.94:0/771391100 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:56.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.055+0000 7fb5b37fe700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5b413fd70 0x7fb5b4142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:56.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.056+0000 7fb5aa7fc700 1 -- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb5b4070c50 msgr2=0x7fb5b4139e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:56.059 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.056+0000 7fb5aa7fc700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb5b4070c50 0x7fb5b4139e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:56.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.056+0000 7fb5aa7fc700 1 -- 172.21.15.94:0/771391100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5b413fd70 msgr2=0x7fb5b4142bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:56.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.056+0000 7fb5aa7fc700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5b413fd70 0x7fb5b4142bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:56.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.056+0000 7fb5aa7fc700 1 -- 172.21.15.94:0/771391100 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a40094f0 con 0x7fb5b40b7160 2023-09-30T00:21:56.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.056+0000 7fb5b2ffd700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb5b4070c50 0x7fb5b4139e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:56.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.056+0000 7fb5aa7fc700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5b40b7160 0x7fb5b40a4660 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00c5d0 tx=0x7fb59c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:56.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.056+0000 7fb5ab7fe700 1 -- 172.21.15.94:0/771391100 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c011590 con 0x7fb5b40b7160 2023-09-30T00:21:56.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.056+0000 7fb5ab7fe700 1 -- 172.21.15.94:0/771391100 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c011710 con 0x7fb5b40b7160 2023-09-30T00:21:56.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.056+0000 7fb5b986b700 1 -- 172.21.15.94:0/771391100 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b413a800 con 0x7fb5b40b7160 2023-09-30T00:21:56.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.056+0000 7fb5ab7fe700 1 -- 172.21.15.94:0/771391100 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c011ad0 con 0x7fb5b40b7160 2023-09-30T00:21:56.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.056+0000 7fb5b986b700 1 -- 172.21.15.94:0/771391100 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b41440d0 con 0x7fb5b40b7160 2023-09-30T00:21:56.060 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.057+0000 7fb5b986b700 1 -- 172.21.15.94:0/771391100 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b41448c0 con 0x7fb5b40b7160 2023-09-30T00:21:56.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.058+0000 7fb5ab7fe700 1 -- 172.21.15.94:0/771391100 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb59c01c460 con 0x7fb5b40b7160 2023-09-30T00:21:56.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.058+0000 7fb5ab7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:56.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.059+0000 7fb5ab7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:56.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.059+0000 7fb5ab7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:56.061 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.059+0000 7fb5ab7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:56.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.059+0000 7fb5ab7fe700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb594098780 0x7fb59409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:56.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.059+0000 7fb5b2ffd700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb594098780 0x7fb59409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:56.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.059+0000 7fb5ab7fe700 1 -- 172.21.15.94:0/771391100 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fb59c0c0a40 con 0x7fb5b40b7160 2023-09-30T00:21:56.062 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.059+0000 7fb5b2ffd700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb594098780 0x7fb59409ac30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40a31a0 tx=0x7fb5a4016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:56.063 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.062+0000 7fb5ab7fe700 1 -- 172.21.15.94:0/771391100 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb59c08e080 con 0x7fb5b40b7160 2023-09-30T00:21:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.285+0000 7fb5b986b700 1 -- 172.21.15.94:0/771391100 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb5b41448c0 con 0x7fb5b40b7160 2023-09-30T00:21:56.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.286+0000 7fb5ab7fe700 1 -- 172.21.15.94:0/771391100 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb5b41448c0 con 0x7fb5b40b7160 2023-09-30T00:21:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:56.286 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.287+0000 7fb5b986b700 10 mgrc shutdown 2023-09-30T00:21:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.287+0000 7fb5b986b700 1 -- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb594098780 msgr2=0x7fb59409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:56.288 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.287+0000 7fb5b986b700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb594098780 0x7fb59409ac30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40a31a0 tx=0x7fb5a4016000 comp rx=0 tx=0).stop 2023-09-30T00:21:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.287+0000 7fb5b986b700 1 -- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5b40b7160 msgr2=0x7fb5b40a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.287+0000 7fb5b986b700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5b40b7160 0x7fb5b40a4660 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00c5d0 tx=0x7fb59c007f60 comp rx=0 tx=0).stop 2023-09-30T00:21:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.288+0000 7fb5b986b700 1 -- 172.21.15.94:0/771391100 shutdown_connections 2023-09-30T00:21:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.288+0000 7fb5b986b700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb594098780 0x7fb59409ac30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.288+0000 7fb5b986b700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb5b4070c50 0x7fb5b4139e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.288+0000 7fb5b986b700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5b413fd70 0x7fb5b4142bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.288+0000 7fb5b986b700 1 --2- 172.21.15.94:0/771391100 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5b40b7160 0x7fb5b40a4660 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.288+0000 7fb5b986b700 1 -- 172.21.15.94:0/771391100 >> 172.21.15.94:0/771391100 conn(0x7fb5b406a090 msgr2=0x7fb5b41393c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.288+0000 7fb5b986b700 1 -- 172.21.15.94:0/771391100 shutdown_connections 2023-09-30T00:21:56.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:56.288+0000 7fb5b986b700 1 -- 172.21.15.94:0/771391100 wait complete. 2023-09-30T00:21:59.300 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:21:59.423 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.422+0000 7f258a1b9700 1 Processor -- start 2023-09-30T00:21:59.423 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.422+0000 7f258a1b9700 1 -- start start 2023-09-30T00:21:59.425 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f258a1b9700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2584070c50 0x7f25840a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:59.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f258a1b9700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f258413fd70 0x7f2584140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:59.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f258a1b9700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25840a1310 con 0x7f25840b7160 2023-09-30T00:21:59.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f258a1b9700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2584071020 con 0x7f2584070c50 2023-09-30T00:21:59.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f258a1b9700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25840b76a0 con 0x7f258413fd70 2023-09-30T00:21:59.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f25837fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2584070c50 0x7f25840a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:59.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f2582ffd700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f25840b7160 0x7f25840a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39556/0 (socket says 172.21.15.94:39556) 2023-09-30T00:21:59.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f2582ffd700 1 -- 172.21.15.94:0/1923156045 learned_addr learned my addr 172.21.15.94:0/1923156045 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:21:59.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f25837fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2584070c50 0x7f25840a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47414/0 (socket says 172.21.15.94:47414) 2023-09-30T00:21:59.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f2583fff700 1 --2- 172.21.15.94:0/1923156045 >> v2:172.21.15.121:3300/0 conn(0x7f258413fd70 0x7f2584140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:59.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f25837fe700 1 -- 172.21.15.94:0/1923156045 >> v2:172.21.15.121:3300/0 conn(0x7f258413fd70 msgr2=0x7f2584140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:21:59.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f25837fe700 1 --2- 172.21.15.94:0/1923156045 >> v2:172.21.15.121:3300/0 conn(0x7f258413fd70 0x7f2584140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:59.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f25837fe700 1 -- 172.21.15.94:0/1923156045 >> v1:172.21.15.94:6789/0 conn(0x7f25840b7160 legacy=0x7f25840a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:59.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f25827fc700 1 -- 172.21.15.94:0/1923156045 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140923197 0 0) 0x7f2578001110 con 0x7f25840b7160 2023-09-30T00:21:59.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f25837fe700 1 -- 172.21.15.94:0/1923156045 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25840b7940 con 0x7f2584070c50 2023-09-30T00:21:59.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.423+0000 7f25837fe700 1 --2- 172.21.15.94:0/1923156045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2584070c50 0x7f25840a0dd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f25840a31a0 tx=0x7f25740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=165f3660305971ff server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:59.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.424+0000 7f25827fc700 1 -- 172.21.15.94:0/1923156045 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257400b690 con 0x7f2584070c50 2023-09-30T00:21:59.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.424+0000 7f25827fc700 1 -- 172.21.15.94:0/1923156045 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257400b7f0 con 0x7f2584070c50 2023-09-30T00:21:59.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.424+0000 7f25827fc700 1 -- 172.21.15.94:0/1923156045 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257400bb90 con 0x7f2584070c50 2023-09-30T00:21:59.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.424+0000 7f258a1b9700 1 -- 172.21.15.94:0/1923156045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2584070c50 msgr2=0x7f25840a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:59.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.424+0000 7f258a1b9700 1 --2- 172.21.15.94:0/1923156045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2584070c50 0x7f25840a0dd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f25840a31a0 tx=0x7f25740091a0 comp rx=0 tx=0).stop 2023-09-30T00:21:59.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.424+0000 7f258a1b9700 1 -- 172.21.15.94:0/1923156045 shutdown_connections 2023-09-30T00:21:59.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.424+0000 7f258a1b9700 1 --2- 172.21.15.94:0/1923156045 >> v2:172.21.15.121:3300/0 conn(0x7f258413fd70 0x7f2584140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:59.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.424+0000 7f258a1b9700 1 --2- 172.21.15.94:0/1923156045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2584070c50 0x7f25840a0dd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:59.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.424+0000 7f258a1b9700 1 -- 172.21.15.94:0/1923156045 >> 172.21.15.94:0/1923156045 conn(0x7f258406a090 msgr2=0x7f25840702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:59.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.424+0000 7f258a1b9700 1 -- 172.21.15.94:0/1923156045 shutdown_connections 2023-09-30T00:21:59.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.424+0000 7f258a1b9700 1 -- 172.21.15.94:0/1923156045 wait complete. 2023-09-30T00:21:59.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.425+0000 7f258a1b9700 1 Processor -- start 2023-09-30T00:21:59.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.425+0000 7f258a1b9700 1 -- start start 2023-09-30T00:21:59.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f258a1b9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2584070c50 0x7f25840a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:59.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f258a1b9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25840b7160 0x7f2584135cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:59.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f258a1b9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f258413fd70 0x7f25840a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:59.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f258a1b9700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25840a47f0 con 0x7f25840b7160 2023-09-30T00:21:59.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f258a1b9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25840a55f0 con 0x7f258413fd70 2023-09-30T00:21:59.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f258a1b9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25840a49e0 con 0x7f2584070c50 2023-09-30T00:21:59.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f2582ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25840b7160 0x7f2584135cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:59.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f25837fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2584070c50 0x7f25840a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:59.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f2582ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25840b7160 0x7f2584135cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47420/0 (socket says 172.21.15.94:47420) 2023-09-30T00:21:59.431 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f2582ffd700 1 -- 172.21.15.94:0/2451629447 learned_addr learned my addr 172.21.15.94:0/2451629447 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:21:59.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f2583fff700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f258413fd70 0x7f25840a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:59.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f2582ffd700 1 -- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2584070c50 msgr2=0x7f25840a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:59.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f2582ffd700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2584070c50 0x7f25840a6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:59.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f2582ffd700 1 -- 172.21.15.94:0/2451629447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f258413fd70 msgr2=0x7f25840a42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:59.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f2582ffd700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f258413fd70 0x7f25840a42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:59.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f2582ffd700 1 -- 172.21.15.94:0/2451629447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2574009420 con 0x7f25840b7160 2023-09-30T00:21:59.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f25837fe700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2584070c50 0x7f25840a6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:21:59.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f2582ffd700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25840b7160 0x7f2584135cf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f257800c810 tx=0x7f2578007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:59.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.426+0000 7f256b7fe700 1 -- 172.21.15.94:0/2451629447 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578005b30 con 0x7f25840b7160 2023-09-30T00:21:59.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.427+0000 7f256b7fe700 1 -- 172.21.15.94:0/2451629447 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2578005c90 con 0x7f25840b7160 2023-09-30T00:21:59.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.427+0000 7f256b7fe700 1 -- 172.21.15.94:0/2451629447 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25780072e0 con 0x7f25840b7160 2023-09-30T00:21:59.432 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.427+0000 7f258a1b9700 1 -- 172.21.15.94:0/2451629447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25840a4c80 con 0x7f25840b7160 2023-09-30T00:21:59.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.427+0000 7f258a1b9700 1 -- 172.21.15.94:0/2451629447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25840a5dd0 con 0x7f25840b7160 2023-09-30T00:21:59.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.427+0000 7f258a1b9700 1 -- 172.21.15.94:0/2451629447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2584071630 con 0x7f25840b7160 2023-09-30T00:21:59.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.428+0000 7f256b7fe700 1 -- 172.21.15.94:0/2451629447 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f2578007690 con 0x7f25840b7160 2023-09-30T00:21:59.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.428+0000 7f256b7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:21:59.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.429+0000 7f256b7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:21:59.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.429+0000 7f256b7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:59.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.429+0000 7f256b7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:21:59.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.429+0000 7f256b7fe700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2570098780 0x7f257009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:21:59.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.429+0000 7f25837fe700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2570098780 0x7f257009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:21:59.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.429+0000 7f256b7fe700 1 -- 172.21.15.94:0/2451629447 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f25780bf730 con 0x7f25840b7160 2023-09-30T00:21:59.433 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.429+0000 7f25837fe700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2570098780 0x7f257009ac30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f25740093f0 tx=0x7f2574017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:21:59.434 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.433+0000 7f256b7fe700 1 -- 172.21.15.94:0/2451629447 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f257804b5a0 con 0x7f25840b7160 2023-09-30T00:21:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.654+0000 7f258a1b9700 1 -- 172.21.15.94:0/2451629447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f25840a1310 con 0x7f25840b7160 2023-09-30T00:21:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.654+0000 7f256b7fe700 1 -- 172.21.15.94:0/2451629447 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2578017100 con 0x7f25840b7160 2023-09-30T00:21:59.655 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:21:59.655 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:21:59.657 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 10 mgrc shutdown 2023-09-30T00:21:59.657 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 1 -- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2570098780 msgr2=0x7f257009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:59.657 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2570098780 0x7f257009ac30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f25740093f0 tx=0x7f2574017000 comp rx=0 tx=0).stop 2023-09-30T00:21:59.657 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 1 -- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25840b7160 msgr2=0x7f2584135cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:21:59.657 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25840b7160 0x7f2584135cf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f257800c810 tx=0x7f2578007c50 comp rx=0 tx=0).stop 2023-09-30T00:21:59.657 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 1 -- 172.21.15.94:0/2451629447 shutdown_connections 2023-09-30T00:21:59.657 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f2570098780 0x7f257009ac30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:59.657 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2584070c50 0x7f25840a6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:59.658 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f258413fd70 0x7f25840a42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:59.658 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 1 --2- 172.21.15.94:0/2451629447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25840b7160 0x7f2584135cf0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:21:59.658 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 1 -- 172.21.15.94:0/2451629447 >> 172.21.15.94:0/2451629447 conn(0x7f258406a090 msgr2=0x7f2584139320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:21:59.658 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 1 -- 172.21.15.94:0/2451629447 shutdown_connections 2023-09-30T00:21:59.658 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:21:59.656+0000 7f258a1b9700 1 -- 172.21.15.94:0/2451629447 wait complete. 2023-09-30T00:22:02.669 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:02.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.793+0000 7f2a1013d700 1 Processor -- start 2023-09-30T00:22:02.794 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.793+0000 7f2a1013d700 1 -- start start 2023-09-30T00:22:02.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a1013d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 0x7f2a080a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:02.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a1013d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2a0813fd70 0x7f2a081401f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:02.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a1013d700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a080a1310 con 0x7f2a08070c50 2023-09-30T00:22:02.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a1013d700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a08071020 con 0x7f2a0813fd70 2023-09-30T00:22:02.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a1013d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a080b76a0 con 0x7f2a080b7160 2023-09-30T00:22:02.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a0d6d8700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 0x7f2a080a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:02.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a0e6da700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2a0813fd70 0x7f2a081401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:02.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a0d6d8700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 0x7f2a080a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:33514/0 (socket says 172.21.15.94:33514) 2023-09-30T00:22:02.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a0d6d8700 1 -- 172.21.15.94:0/924773338 learned_addr learned my addr 172.21.15.94:0/924773338 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:02.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a0e6da700 1 -- 172.21.15.94:0/924773338 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 msgr2=0x7f2a080a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:02.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a0e6da700 1 --2- 172.21.15.94:0/924773338 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 0x7f2a080a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:02.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a0e6da700 1 -- 172.21.15.94:0/924773338 >> v1:172.21.15.94:6789/0 conn(0x7f2a08070c50 legacy=0x7f2a080a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:02.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a0e6da700 1 -- 172.21.15.94:0/924773338 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a080b7940 con 0x7f2a0813fd70 2023-09-30T00:22:02.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a0ced7700 1 -- 172.21.15.94:0/924773338 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202094806 0 0) 0x7f2a04001110 con 0x7f2a08070c50 2023-09-30T00:22:02.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.794+0000 7f2a0d6d8700 1 --2- 172.21.15.94:0/924773338 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 0x7f2a080a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:02.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.795+0000 7f2a0e6da700 1 --2- 172.21.15.94:0/924773338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a0813fd70 0x7f2a081401f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009e60 tx=0x7f2a00009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0857c4e95703a1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:02.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.795+0000 7f2a0ced7700 1 -- 172.21.15.94:0/924773338 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0000e070 con 0x7f2a0813fd70 2023-09-30T00:22:02.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.795+0000 7f2a0ced7700 1 -- 172.21.15.94:0/924773338 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0000bbf0 con 0x7f2a0813fd70 2023-09-30T00:22:02.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.795+0000 7f2a0ced7700 1 -- 172.21.15.94:0/924773338 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00012430 con 0x7f2a0813fd70 2023-09-30T00:22:02.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.795+0000 7f2a1013d700 1 -- 172.21.15.94:0/924773338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a0813fd70 msgr2=0x7f2a081401f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:02.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.795+0000 7f2a1013d700 1 --2- 172.21.15.94:0/924773338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a0813fd70 0x7f2a081401f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009e60 tx=0x7f2a00009780 comp rx=0 tx=0).stop 2023-09-30T00:22:02.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.795+0000 7f2a1013d700 1 -- 172.21.15.94:0/924773338 shutdown_connections 2023-09-30T00:22:02.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.795+0000 7f2a1013d700 1 --2- 172.21.15.94:0/924773338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a0813fd70 0x7f2a081401f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:02.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.795+0000 7f2a1013d700 1 --2- 172.21.15.94:0/924773338 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 0x7f2a080a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:02.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.795+0000 7f2a1013d700 1 -- 172.21.15.94:0/924773338 >> 172.21.15.94:0/924773338 conn(0x7f2a0806a090 msgr2=0x7f2a080702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:02.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.795+0000 7f2a1013d700 1 -- 172.21.15.94:0/924773338 shutdown_connections 2023-09-30T00:22:02.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.795+0000 7f2a1013d700 1 -- 172.21.15.94:0/924773338 wait complete. 2023-09-30T00:22:02.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.796+0000 7f2a1013d700 1 Processor -- start 2023-09-30T00:22:02.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.796+0000 7f2a1013d700 1 -- start start 2023-09-30T00:22:02.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a1013d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a08070c50 0x7f2a08139c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:02.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a1013d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 0x7f2a080a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:02.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a1013d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a0813fd70 0x7f2a080a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:02.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a1013d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a080a4ef0 con 0x7f2a08070c50 2023-09-30T00:22:02.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a1013d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a08144890 con 0x7f2a0813fd70 2023-09-30T00:22:02.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a1013d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a080a6d10 con 0x7f2a080b7160 2023-09-30T00:22:02.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a0ded9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a08070c50 0x7f2a08139c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:02.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a0d6d8700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 0x7f2a080a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:02.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a0ded9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a08070c50 0x7f2a08139c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47446/0 (socket says 172.21.15.94:47446) 2023-09-30T00:22:02.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a0ded9700 1 -- 172.21.15.94:0/1085887404 learned_addr learned my addr 172.21.15.94:0/1085887404 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:02.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a0d6d8700 1 -- 172.21.15.94:0/1085887404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a0813fd70 msgr2=0x7f2a080a49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:02.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a0d6d8700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a0813fd70 0x7f2a080a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:02.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a0d6d8700 1 -- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a08070c50 msgr2=0x7f2a08139c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:02.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a0d6d8700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a08070c50 0x7f2a08139c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:02.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a0d6d8700 1 -- 172.21.15.94:0/1085887404 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f80094f0 con 0x7f2a080b7160 2023-09-30T00:22:02.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a0ded9700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a08070c50 0x7f2a08139c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:02.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f2a0d6d8700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 0x7f2a080a4470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f29f8006fd0 tx=0x7f29f8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:02.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f29fdffb700 1 -- 172.21.15.94:0/1085887404 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f800c850 con 0x7f2a080b7160 2023-09-30T00:22:02.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.797+0000 7f29fdffb700 1 -- 172.21.15.94:0/1085887404 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f800c9b0 con 0x7f2a080b7160 2023-09-30T00:22:02.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.798+0000 7f2a1013d700 1 -- 172.21.15.94:0/1085887404 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a00009a20 con 0x7f2a080b7160 2023-09-30T00:22:02.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.798+0000 7f29fdffb700 1 -- 172.21.15.94:0/1085887404 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f800cd30 con 0x7f2a080b7160 2023-09-30T00:22:02.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.798+0000 7f2a1013d700 1 -- 172.21.15.94:0/1085887404 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a081440d0 con 0x7f2a080b7160 2023-09-30T00:22:02.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.799+0000 7f2a1013d700 1 -- 172.21.15.94:0/1085887404 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a08144a20 con 0x7f2a080b7160 2023-09-30T00:22:02.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.800+0000 7f29fdffb700 1 -- 172.21.15.94:0/1085887404 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f29f8013680 con 0x7f2a080b7160 2023-09-30T00:22:02.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.800+0000 7f29fdffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:02.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.800+0000 7f29fdffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:02.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.800+0000 7f29fdffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:02.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.800+0000 7f29fdffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:02.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.800+0000 7f29fdffb700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f29ec098780 0x7f29ec09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:02.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.800+0000 7f29fdffb700 1 -- 172.21.15.94:0/1085887404 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f29f80c06e0 con 0x7f2a080b7160 2023-09-30T00:22:02.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.800+0000 7f2a0ded9700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f29ec098780 0x7f29ec09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:02.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.801+0000 7f2a0ded9700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f29ec098780 0x7f29ec09ac30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2a04001370 tx=0x7f2a04009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:02.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:02.804+0000 7f29fdffb700 1 -- 172.21.15.94:0/1085887404 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a08144a20 con 0x7f2a080b7160 2023-09-30T00:22:03.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.029+0000 7f2a1013d700 1 -- 172.21.15.94:0/1085887404 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2a081449d0 con 0x7f2a080b7160 2023-09-30T00:22:03.030 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.029+0000 7f29fdffb700 1 -- 172.21.15.94:0/1085887404 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f29f808dd20 con 0x7f2a080b7160 2023-09-30T00:22:03.030 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:03.030 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:03.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 10 mgrc shutdown 2023-09-30T00:22:03.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 1 -- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f29ec098780 msgr2=0x7f29ec09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:03.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f29ec098780 0x7f29ec09ac30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2a04001370 tx=0x7f2a04009040 comp rx=0 tx=0).stop 2023-09-30T00:22:03.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 1 -- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 msgr2=0x7f2a080a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:03.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 0x7f2a080a4470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f29f8006fd0 tx=0x7f29f8009250 comp rx=0 tx=0).stop 2023-09-30T00:22:03.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 1 -- 172.21.15.94:0/1085887404 shutdown_connections 2023-09-30T00:22:03.032 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f29ec098780 0x7f29ec09ac30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a0813fd70 0x7f2a080a49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a08070c50 0x7f2a08139c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 1 --2- 172.21.15.94:0/1085887404 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2a080b7160 0x7f2a080a4470 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 1 -- 172.21.15.94:0/1085887404 >> 172.21.15.94:0/1085887404 conn(0x7f2a0806a090 msgr2=0x7f2a081391b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 1 -- 172.21.15.94:0/1085887404 shutdown_connections 2023-09-30T00:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:03.031+0000 7f2a1013d700 1 -- 172.21.15.94:0/1085887404 wait complete. 2023-09-30T00:22:06.044 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:06.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.169+0000 7fb5defd9700 1 Processor -- start 2023-09-30T00:22:06.170 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.169+0000 7fb5defd9700 1 -- start start 2023-09-30T00:22:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5defd9700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb5d80b6e50 0x7fb5d80a1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:06.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5defd9700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb5d813fe20 0x7fb5d8140260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:06.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5defd9700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d80b7220 con 0x7fb5d80a2140 2023-09-30T00:22:06.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5defd9700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d81407a0 con 0x7fb5d80b6e50 2023-09-30T00:22:06.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5defd9700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d80a36c0 con 0x7fb5d813fe20 2023-09-30T00:22:06.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5ddfd7700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb5d80b6e50 0x7fb5d80a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:06.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5dd7d6700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb5d80a2140 0x7fb5d80a2fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43858/0 (socket says 172.21.15.94:43858) 2023-09-30T00:22:06.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5dd7d6700 1 -- 172.21.15.94:0/2677065278 learned_addr learned my addr 172.21.15.94:0/2677065278 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:22:06.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5de7d8700 1 --2- 172.21.15.94:0/2677065278 >> v2:172.21.15.121:3300/0 conn(0x7fb5d813fe20 0x7fb5d8140260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:06.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5ddfd7700 1 -- 172.21.15.94:0/2677065278 >> v2:172.21.15.121:3300/0 conn(0x7fb5d813fe20 msgr2=0x7fb5d8140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:06.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5ddfd7700 1 --2- 172.21.15.94:0/2677065278 >> v2:172.21.15.121:3300/0 conn(0x7fb5d813fe20 0x7fb5d8140260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:06.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5ddfd7700 1 -- 172.21.15.94:0/2677065278 >> v1:172.21.15.94:6789/0 conn(0x7fb5d80a2140 legacy=0x7fb5d80a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:06.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5ddfd7700 1 -- 172.21.15.94:0/2677065278 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d80a2870 con 0x7fb5d80b6e50 2023-09-30T00:22:06.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5dcfd5700 1 -- 172.21.15.94:0/2677065278 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005874512 0 0) 0x7fb5d80b7220 con 0x7fb5d80a2140 2023-09-30T00:22:06.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.170+0000 7fb5de7d8700 1 --2- 172.21.15.94:0/2677065278 >> v2:172.21.15.121:3300/0 conn(0x7fb5d813fe20 0x7fb5d8140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:22:06.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.171+0000 7fb5ddfd7700 1 --2- 172.21.15.94:0/2677065278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5d80b6e50 0x7fb5d80a1080 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb5d80a0d50 tx=0x7fb5d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5708d1b6ac09357 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:06.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.171+0000 7fb5dcfd5700 1 -- 172.21.15.94:0/2677065278 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d400b6b0 con 0x7fb5d80b6e50 2023-09-30T00:22:06.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.171+0000 7fb5dcfd5700 1 -- 172.21.15.94:0/2677065278 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d400b810 con 0x7fb5d80b6e50 2023-09-30T00:22:06.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.171+0000 7fb5dcfd5700 1 -- 172.21.15.94:0/2677065278 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d400bbb0 con 0x7fb5d80b6e50 2023-09-30T00:22:06.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.171+0000 7fb5defd9700 1 -- 172.21.15.94:0/2677065278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5d80b6e50 msgr2=0x7fb5d80a1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:06.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.171+0000 7fb5defd9700 1 --2- 172.21.15.94:0/2677065278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5d80b6e50 0x7fb5d80a1080 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb5d80a0d50 tx=0x7fb5d40091a0 comp rx=0 tx=0).stop 2023-09-30T00:22:06.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.171+0000 7fb5defd9700 1 -- 172.21.15.94:0/2677065278 shutdown_connections 2023-09-30T00:22:06.175 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.171+0000 7fb5defd9700 1 --2- 172.21.15.94:0/2677065278 >> v2:172.21.15.121:3300/0 conn(0x7fb5d813fe20 0x7fb5d8140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:06.175 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.171+0000 7fb5defd9700 1 --2- 172.21.15.94:0/2677065278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5d80b6e50 0x7fb5d80a1080 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:06.175 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.171+0000 7fb5defd9700 1 -- 172.21.15.94:0/2677065278 >> 172.21.15.94:0/2677065278 conn(0x7fb5d80702f0 msgr2=0x7fb5d80706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:06.175 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.171+0000 7fb5defd9700 1 -- 172.21.15.94:0/2677065278 shutdown_connections 2023-09-30T00:22:06.175 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.171+0000 7fb5defd9700 1 -- 172.21.15.94:0/2677065278 wait complete. 2023-09-30T00:22:06.175 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.172+0000 7fb5defd9700 1 Processor -- start 2023-09-30T00:22:06.175 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.172+0000 7fb5defd9700 1 -- start start 2023-09-30T00:22:06.177 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5defd9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5d80a2140 0x7fb5d81441b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:06.178 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5defd9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb5d80b6e50 0x7fb5d8141c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:06.178 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5defd9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5d813fe20 0x7fb5d813b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:06.178 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5defd9700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d8142570 con 0x7fb5d80a2140 2023-09-30T00:22:06.178 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5defd9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d8142830 con 0x7fb5d813fe20 2023-09-30T00:22:06.178 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5defd9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d8144910 con 0x7fb5d80b6e50 2023-09-30T00:22:06.178 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5ddfd7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5d80a2140 0x7fb5d81441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:06.178 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5ddfd7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5d80a2140 0x7fb5d81441b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57510/0 (socket says 172.21.15.94:57510) 2023-09-30T00:22:06.178 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5ddfd7700 1 -- 172.21.15.94:0/2727224558 learned_addr learned my addr 172.21.15.94:0/2727224558 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:06.178 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5dd7d6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb5d80b6e50 0x7fb5d8141c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:06.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5de7d8700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5d813fe20 0x7fb5d813b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:06.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5ddfd7700 1 -- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb5d80b6e50 msgr2=0x7fb5d8141c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:06.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5ddfd7700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb5d80b6e50 0x7fb5d8141c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:06.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5ddfd7700 1 -- 172.21.15.94:0/2727224558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5d813fe20 msgr2=0x7fb5d813b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:06.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5ddfd7700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5d813fe20 0x7fb5d813b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:06.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5ddfd7700 1 -- 172.21.15.94:0/2727224558 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d4009420 con 0x7fb5d80a2140 2023-09-30T00:22:06.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.173+0000 7fb5de7d8700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5d813fe20 0x7fb5d813b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:22:06.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.174+0000 7fb5ddfd7700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5d80a2140 0x7fb5d81441b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8071280 tx=0x7fb5d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:06.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.174+0000 7fb5cdffb700 1 -- 172.21.15.94:0/2727224558 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d400bb90 con 0x7fb5d80a2140 2023-09-30T00:22:06.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.174+0000 7fb5cdffb700 1 -- 172.21.15.94:0/2727224558 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d4011420 con 0x7fb5d80a2140 2023-09-30T00:22:06.179 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.174+0000 7fb5defd9700 1 -- 172.21.15.94:0/2727224558 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d81421d0 con 0x7fb5d80a2140 2023-09-30T00:22:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.174+0000 7fb5cdffb700 1 -- 172.21.15.94:0/2727224558 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d40117a0 con 0x7fb5d80a2140 2023-09-30T00:22:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.174+0000 7fb5defd9700 1 -- 172.21.15.94:0/2727224558 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d80a57a0 con 0x7fb5d80a2140 2023-09-30T00:22:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.175+0000 7fb5defd9700 1 -- 172.21.15.94:0/2727224558 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d80a65e0 con 0x7fb5d80a2140 2023-09-30T00:22:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.176+0000 7fb5cdffb700 1 -- 172.21.15.94:0/2727224558 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fb5d4011b90 con 0x7fb5d80a2140 2023-09-30T00:22:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.176+0000 7fb5cdffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.176+0000 7fb5cdffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.176+0000 7fb5cdffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.176+0000 7fb5cdffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.176+0000 7fb5cdffb700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb5bc0987b0 0x7fb5bc09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.176+0000 7fb5cdffb700 1 -- 172.21.15.94:0/2727224558 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fb5d40c20b0 con 0x7fb5d80a2140 2023-09-30T00:22:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.176+0000 7fb5dd7d6700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb5bc0987b0 0x7fb5bc09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:06.180 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.177+0000 7fb5dd7d6700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb5bc0987b0 0x7fb5bc09ac60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8143ce0 tx=0x7fb5c8007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:06.181 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.180+0000 7fb5cdffb700 1 -- 172.21.15.94:0/2727224558 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb5d408f780 con 0x7fb5d80a2140 2023-09-30T00:22:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.404+0000 7fb5defd9700 1 -- 172.21.15.94:0/2727224558 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb5d8066e50 con 0x7fb5d80a2140 2023-09-30T00:22:06.406 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.405+0000 7fb5cdffb700 1 -- 172.21.15.94:0/2727224558 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb5d4092da0 con 0x7fb5d80a2140 2023-09-30T00:22:06.406 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:06.406 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:06.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 10 mgrc shutdown 2023-09-30T00:22:06.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 1 -- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb5bc0987b0 msgr2=0x7fb5bc09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:06.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb5bc0987b0 0x7fb5bc09ac60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8143ce0 tx=0x7fb5c8007d30 comp rx=0 tx=0).stop 2023-09-30T00:22:06.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 1 -- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5d80a2140 msgr2=0x7fb5d81441b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:06.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5d80a2140 0x7fb5d81441b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8071280 tx=0x7fb5d4017000 comp rx=0 tx=0).stop 2023-09-30T00:22:06.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 1 -- 172.21.15.94:0/2727224558 shutdown_connections 2023-09-30T00:22:06.408 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fb5bc0987b0 0x7fb5bc09ac60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:06.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5d813fe20 0x7fb5d813b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:06.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5d80a2140 0x7fb5d81441b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:06.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 1 --2- 172.21.15.94:0/2727224558 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb5d80b6e50 0x7fb5d8141c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:06.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 1 -- 172.21.15.94:0/2727224558 >> 172.21.15.94:0/2727224558 conn(0x7fb5d80702f0 msgr2=0x7fb5d80a6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:06.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 1 -- 172.21.15.94:0/2727224558 shutdown_connections 2023-09-30T00:22:06.409 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:06.407+0000 7fb5defd9700 1 -- 172.21.15.94:0/2727224558 wait complete. 2023-09-30T00:22:09.420 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.543+0000 7f52e1880700 1 Processor -- start 2023-09-30T00:22:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.543+0000 7f52e1880700 1 -- start start 2023-09-30T00:22:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.543+0000 7f52e1880700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f52dc0a0dd0 0x7f52dc14c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.543+0000 7f52e1880700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52dc071020 con 0x7f52dc0b7160 2023-09-30T00:22:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.543+0000 7f52e1880700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52dc0b76a0 con 0x7f52dc070c50 2023-09-30T00:22:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.543+0000 7f52e1880700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52dc0b7940 con 0x7f52dc0a0dd0 2023-09-30T00:22:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.543+0000 7f52d27fc700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f52dc0b7160 0x7f52dc13fd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43864/0 (socket says 172.21.15.94:43864) 2023-09-30T00:22:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.543+0000 7f52d27fc700 1 -- 172.21.15.94:0/2585868785 learned_addr learned my addr 172.21.15.94:0/2585868785 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:22:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52db7fe700 1 --2- 172.21.15.94:0/2585868785 >> v2:172.21.15.121:3300/0 conn(0x7f52dc0a0dd0 0x7f52dc14c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588225442 0 0) 0x7f52dc071020 con 0x7f52dc0b7160 2023-09-30T00:22:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c8003570 con 0x7f52dc0b7160 2023-09-30T00:22:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856995873 0 0) 0x7f52dc0b76a0 con 0x7f52dc070c50 2023-09-30T00:22:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52dc071020 con 0x7f52dc070c50 2023-09-30T00:22:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4238193121 0 0) 0x7f52c8003570 con 0x7f52dc0b7160 2023-09-30T00:22:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 >> v2:172.21.15.121:3300/0 conn(0x7f52dc0a0dd0 msgr2=0x7f52dc14c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 --2- 172.21.15.94:0/2585868785 >> v2:172.21.15.121:3300/0 conn(0x7f52dc0a0dd0 0x7f52dc14c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 >> v1:172.21.15.121:6789/0 conn(0x7f52dc070c50 legacy=0x7f52dc0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc0a2070 con 0x7f52dc0b7160 2023-09-30T00:22:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3153706020 0 0) 0x7f52c4001470 con 0x7f52dc0b7160 2023-09-30T00:22:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 >> v1:172.21.15.94:6789/0 conn(0x7f52dc0b7160 legacy=0x7f52dc13fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 --2- 172.21.15.94:0/2585868785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52c8003c40 0x7f52c8004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 --2- 172.21.15.94:0/2585868785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52c8004810 0x7f52c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 --2- 172.21.15.94:0/2585868785 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52c8005210 0x7f52c800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc0b7940 con 0x7f52c8003c40 2023-09-30T00:22:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c800c130 con 0x7f52c8004810 2023-09-30T00:22:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c800c2c0 con 0x7f52c8005210 2023-09-30T00:22:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52daffd700 1 --2- 172.21.15.94:0/2585868785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52c8003c40 0x7f52c8004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52d27fc700 1 --2- 172.21.15.94:0/2585868785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52c8004810 0x7f52c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52db7fe700 1 --2- 172.21.15.94:0/2585868785 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52c8005210 0x7f52c800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52daffd700 1 -- 172.21.15.94:0/2585868785 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52c8005210 msgr2=0x7f52c800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52daffd700 1 --2- 172.21.15.94:0/2585868785 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52c8005210 0x7f52c800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52daffd700 1 -- 172.21.15.94:0/2585868785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52c8004810 msgr2=0x7f52c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52daffd700 1 --2- 172.21.15.94:0/2585868785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52c8004810 0x7f52c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52daffd700 1 -- 172.21.15.94:0/2585868785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c8003570 con 0x7f52c8003c40 2023-09-30T00:22:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.544+0000 7f52db7fe700 1 -- 172.21.15.94:0/2585868785 reap_dead start 2023-09-30T00:22:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.545+0000 7f52daffd700 1 --2- 172.21.15.94:0/2585868785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52c8003c40 0x7f52c8004030 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f52cc001a80 tx=0x7f52cc007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77356ac84b1739fb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.545+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc00c550 con 0x7f52c8003c40 2023-09-30T00:22:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.545+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52cc00c6b0 con 0x7f52c8003c40 2023-09-30T00:22:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.545+0000 7f52da7fc700 1 -- 172.21.15.94:0/2585868785 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc00ca50 con 0x7f52c8003c40 2023-09-30T00:22:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.545+0000 7f52e1880700 1 -- 172.21.15.94:0/2585868785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52c8003c40 msgr2=0x7f52c8004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.545+0000 7f52e1880700 1 --2- 172.21.15.94:0/2585868785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52c8003c40 0x7f52c8004030 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f52cc001a80 tx=0x7f52cc007d30 comp rx=0 tx=0).stop 2023-09-30T00:22:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.545+0000 7f52e1880700 1 -- 172.21.15.94:0/2585868785 shutdown_connections 2023-09-30T00:22:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.545+0000 7f52e1880700 1 --2- 172.21.15.94:0/2585868785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52c8003c40 0x7f52c8004030 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.545+0000 7f52e1880700 1 -- 172.21.15.94:0/2585868785 >> 172.21.15.94:0/2585868785 conn(0x7f52dc06a090 msgr2=0x7f52dc0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.545+0000 7f52e1880700 1 -- 172.21.15.94:0/2585868785 shutdown_connections 2023-09-30T00:22:09.551 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.545+0000 7f52e1880700 1 -- 172.21.15.94:0/2585868785 wait complete. 2023-09-30T00:22:09.551 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.546+0000 7f52e1880700 1 Processor -- start 2023-09-30T00:22:09.551 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.546+0000 7f52e1880700 1 -- start start 2023-09-30T00:22:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52e1880700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52c8003c40 0x7f52dc0aaf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52e1880700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52dc0ac470 0x7f52dc0ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52e1880700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52dc1508c0 0x7f52dc150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52e1880700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc0a50d0 con 0x7f52c8003c40 2023-09-30T00:22:09.553 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52e1880700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc1503d0 con 0x7f52dc0ac470 2023-09-30T00:22:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52e1880700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc0acdc0 con 0x7f52dc1508c0 2023-09-30T00:22:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52db7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52dc1508c0 0x7f52dc150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52daffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52c8003c40 0x7f52dc0aaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52db7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52dc1508c0 0x7f52dc150d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:60588/0 (socket says 172.21.15.94:60588) 2023-09-30T00:22:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52db7fe700 1 -- 172.21.15.94:0/1858710651 learned_addr learned my addr 172.21.15.94:0/1858710651 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52d27fc700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52dc0ac470 0x7f52dc0ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52daffd700 1 -- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52dc1508c0 msgr2=0x7f52dc150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52daffd700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52dc1508c0 0x7f52dc150d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52daffd700 1 -- 172.21.15.94:0/1858710651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52dc0ac470 msgr2=0x7f52dc0ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52daffd700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52dc0ac470 0x7f52dc0ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:09.554 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52daffd700 1 -- 172.21.15.94:0/1858710651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc150600 con 0x7f52c8003c40 2023-09-30T00:22:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52db7fe700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52dc1508c0 0x7f52dc150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52daffd700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52c8003c40 0x7f52dc0aaf40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f52cc014000 tx=0x7f52cc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.547+0000 7f52d37fe700 1 -- 172.21.15.94:0/1858710651 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc025070 con 0x7f52c8003c40 2023-09-30T00:22:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.548+0000 7f52d37fe700 1 -- 172.21.15.94:0/1858710651 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52cc0013a0 con 0x7f52c8003c40 2023-09-30T00:22:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.548+0000 7f52e1880700 1 -- 172.21.15.94:0/1858710651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52dc0a53a0 con 0x7f52c8003c40 2023-09-30T00:22:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.548+0000 7f52d37fe700 1 -- 172.21.15.94:0/1858710651 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc012dc0 con 0x7f52c8003c40 2023-09-30T00:22:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.548+0000 7f52e1880700 1 -- 172.21.15.94:0/1858710651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52dc0aa920 con 0x7f52c8003c40 2023-09-30T00:22:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.548+0000 7f52e1880700 1 -- 172.21.15.94:0/1858710651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52dc0aaab0 con 0x7f52c8003c40 2023-09-30T00:22:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.549+0000 7f52d37fe700 1 -- 172.21.15.94:0/1858710651 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f52cc02a410 con 0x7f52c8003c40 2023-09-30T00:22:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.549+0000 7f52d37fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.550+0000 7f52d37fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.550+0000 7f52d37fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:09.556 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.550+0000 7f52d37fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:09.556 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.550+0000 7f52d37fe700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f52bc0987b0 0x7f52bc09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:09.556 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.550+0000 7f52d27fc700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f52bc0987b0 0x7f52bc09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:09.556 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.550+0000 7f52d37fe700 1 -- 172.21.15.94:0/1858710651 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f52cc0c1a40 con 0x7f52c8003c40 2023-09-30T00:22:09.556 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.550+0000 7f52d27fc700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f52bc0987b0 0x7f52bc09ac60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f52c40015a0 tx=0x7f52c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:09.557 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:22:09.557 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.554+0000 7f52d37fe700 1 -- 172.21.15.94:0/1858710651 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f52cc0c3e60 con 0x7f52c8003c40 2023-09-30T00:22:09.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:22:09.781 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.780+0000 7f52e1880700 1 -- 172.21.15.94:0/1858710651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f52dc071630 con 0x7f52c8003c40 2023-09-30T00:22:09.781 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.781+0000 7f52d37fe700 1 -- 172.21.15.94:0/1858710651 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f52cc017080 con 0x7f52c8003c40 2023-09-30T00:22:09.782 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:09.782 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:09.784 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 10 mgrc shutdown 2023-09-30T00:22:09.784 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 1 -- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f52bc0987b0 msgr2=0x7f52bc09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:09.784 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f52bc0987b0 0x7f52bc09ac60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f52c40015a0 tx=0x7f52c4009040 comp rx=0 tx=0).stop 2023-09-30T00:22:09.784 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 1 -- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52c8003c40 msgr2=0x7f52dc0aaf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:09.784 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52c8003c40 0x7f52dc0aaf40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f52cc014000 tx=0x7f52cc015000 comp rx=0 tx=0).stop 2023-09-30T00:22:09.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 1 -- 172.21.15.94:0/1858710651 shutdown_connections 2023-09-30T00:22:09.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f52bc0987b0 0x7f52bc09ac60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:09.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52dc1508c0 0x7f52dc150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:09.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52c8003c40 0x7f52dc0aaf40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:09.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 1 --2- 172.21.15.94:0/1858710651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52dc0ac470 0x7f52dc0ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:09.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 1 -- 172.21.15.94:0/1858710651 >> 172.21.15.94:0/1858710651 conn(0x7f52dc06a090 msgr2=0x7f52dc068500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:09.785 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 1 -- 172.21.15.94:0/1858710651 shutdown_connections 2023-09-30T00:22:09.786 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:09.783+0000 7f52e1880700 1 -- 172.21.15.94:0/1858710651 wait complete. 2023-09-30T00:22:12.796 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:12.921 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.920+0000 7f16b54f3700 1 Processor -- start 2023-09-30T00:22:12.921 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.920+0000 7f16b54f3700 1 -- start start 2023-09-30T00:22:12.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16b54f3700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f16b0067a60 0x7f16b00b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:12.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16b54f3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b00a6070 0x7f16b00a5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:12.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16b54f3700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b0067e30 con 0x7f16b00b5440 2023-09-30T00:22:12.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16b54f3700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b01419a0 con 0x7f16b0067a60 2023-09-30T00:22:12.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16b54f3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b00a5c60 con 0x7f16b00a6070 2023-09-30T00:22:12.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16a67fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b00a6070 0x7f16b00a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:12.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16a67fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b00a6070 0x7f16b00a5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:60596/0 (socket says 172.21.15.94:60596) 2023-09-30T00:22:12.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16a67fc700 1 -- 172.21.15.94:0/563567208 learned_addr learned my addr 172.21.15.94:0/563567208 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:12.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16a67fc700 1 -- 172.21.15.94:0/563567208 >> v2:172.21.15.121:3300/0 conn(0x7f16b0067a60 msgr2=0x7f16b00b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:12.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16aeffd700 1 --2- 172.21.15.94:0/563567208 >> v2:172.21.15.121:3300/0 conn(0x7f16b0067a60 0x7f16b00b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:12.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16a67fc700 1 --2- 172.21.15.94:0/563567208 >> v2:172.21.15.121:3300/0 conn(0x7f16b0067a60 0x7f16b00b4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:12.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16a67fc700 1 -- 172.21.15.94:0/563567208 >> v1:172.21.15.121:6789/0 conn(0x7f16b00b5440 legacy=0x7f16b00a6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:12.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.921+0000 7f16a67fc700 1 -- 172.21.15.94:0/563567208 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b00a6570 con 0x7f16b00a6070 2023-09-30T00:22:12.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.922+0000 7f16a67fc700 1 --2- 172.21.15.94:0/563567208 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b00a6070 0x7f16b00a5720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f16980099b0 tx=0x7f1698009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faa9999a3ea4496a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:12.924 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.922+0000 7f16ae7fc700 1 -- 172.21.15.94:0/563567208 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169800b820 con 0x7f16b00a6070 2023-09-30T00:22:12.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.922+0000 7f16ae7fc700 1 -- 172.21.15.94:0/563567208 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169800b980 con 0x7f16b00a6070 2023-09-30T00:22:12.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.922+0000 7f16ae7fc700 1 -- 172.21.15.94:0/563567208 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169800bd20 con 0x7f16b00a6070 2023-09-30T00:22:12.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.922+0000 7f16b54f3700 1 -- 172.21.15.94:0/563567208 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b00a6070 msgr2=0x7f16b00a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:12.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.922+0000 7f16b54f3700 1 --2- 172.21.15.94:0/563567208 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b00a6070 0x7f16b00a5720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f16980099b0 tx=0x7f1698009250 comp rx=0 tx=0).stop 2023-09-30T00:22:12.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.922+0000 7f16b54f3700 1 -- 172.21.15.94:0/563567208 shutdown_connections 2023-09-30T00:22:12.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.922+0000 7f16b54f3700 1 --2- 172.21.15.94:0/563567208 >> v2:172.21.15.121:3300/0 conn(0x7f16b0067a60 0x7f16b00b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:12.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.922+0000 7f16b54f3700 1 --2- 172.21.15.94:0/563567208 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b00a6070 0x7f16b00a5720 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:12.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.922+0000 7f16b54f3700 1 -- 172.21.15.94:0/563567208 >> 172.21.15.94:0/563567208 conn(0x7f16b006cb10 msgr2=0x7f16b006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:12.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.922+0000 7f16b54f3700 1 -- 172.21.15.94:0/563567208 shutdown_connections 2023-09-30T00:22:12.925 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.922+0000 7f16b54f3700 1 -- 172.21.15.94:0/563567208 wait complete. 2023-09-30T00:22:12.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.923+0000 7f16b54f3700 1 Processor -- start 2023-09-30T00:22:12.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.923+0000 7f16b54f3700 1 -- start start 2023-09-30T00:22:12.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16b54f3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b0067a60 0x7f16b00a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:12.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16b54f3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b00a6070 0x7f16b00a41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:12.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16b54f3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b00b5440 0x7f16b0154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:12.928 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16b54f3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b00a1430 con 0x7f16b00a6070 2023-09-30T00:22:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16b54f3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b0144190 con 0x7f16b00b5440 2023-09-30T00:22:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16b54f3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b01507d0 con 0x7f16b0067a60 2023-09-30T00:22:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16aeffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b0067a60 0x7f16b00a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16a67fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b00a6070 0x7f16b00a41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16aeffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b0067a60 0x7f16b00a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:60604/0 (socket says 172.21.15.94:60604) 2023-09-30T00:22:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16aeffd700 1 -- 172.21.15.94:0/2054124208 learned_addr learned my addr 172.21.15.94:0/2054124208 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16af7fe700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b00b5440 0x7f16b0154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16aeffd700 1 -- 172.21.15.94:0/2054124208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b00b5440 msgr2=0x7f16b0154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16aeffd700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b00b5440 0x7f16b0154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16aeffd700 1 -- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b00a6070 msgr2=0x7f16b00a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16aeffd700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b00a6070 0x7f16b00a41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16aeffd700 1 -- 172.21.15.94:0/2054124208 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16980094f0 con 0x7f16b0067a60 2023-09-30T00:22:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.924+0000 7f16a67fc700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b00a6070 0x7f16b00a41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.925+0000 7f16aeffd700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b0067a60 0x7f16b00a0ec0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f16a000c7d0 tx=0x7f16a0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.925+0000 7f16a77fe700 1 -- 172.21.15.94:0/2054124208 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0011600 con 0x7f16b0067a60 2023-09-30T00:22:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.925+0000 7f16a77fe700 1 -- 172.21.15.94:0/2054124208 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a0011760 con 0x7f16b0067a60 2023-09-30T00:22:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.925+0000 7f16a77fe700 1 -- 172.21.15.94:0/2054124208 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0011b20 con 0x7f16b0067a60 2023-09-30T00:22:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.925+0000 7f16b54f3700 1 -- 172.21.15.94:0/2054124208 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b00a4700 con 0x7f16b0067a60 2023-09-30T00:22:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.925+0000 7f16b54f3700 1 -- 172.21.15.94:0/2054124208 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b01546a0 con 0x7f16b0067a60 2023-09-30T00:22:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.926+0000 7f16b54f3700 1 -- 172.21.15.94:0/2054124208 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b0071630 con 0x7f16b0067a60 2023-09-30T00:22:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.927+0000 7f16a77fe700 1 -- 172.21.15.94:0/2054124208 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f16a00193f0 con 0x7f16b0067a60 2023-09-30T00:22:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.927+0000 7f16a77fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.927+0000 7f16a77fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.927+0000 7f16a77fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.927+0000 7f16a77fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.927+0000 7f16a77fe700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1690098780 0x7f169009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.927+0000 7f16a77fe700 1 -- 172.21.15.94:0/2054124208 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f16a00c07f0 con 0x7f16b0067a60 2023-09-30T00:22:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.927+0000 7f16a67fc700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1690098780 0x7f169009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.928+0000 7f16a67fc700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1690098780 0x7f169009ac30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f16980099b0 tx=0x7f1698016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:12.931+0000 7f16a77fe700 1 -- 172.21.15.94:0/2054124208 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f16a008de30 con 0x7f16b0067a60 2023-09-30T00:22:13.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.152+0000 7f16b54f3700 1 -- 172.21.15.94:0/2054124208 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f16b0153da0 con 0x7f16b0067a60 2023-09-30T00:22:13.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.153+0000 7f16a77fe700 1 -- 172.21.15.94:0/2054124208 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f16a0091450 con 0x7f16b0067a60 2023-09-30T00:22:13.154 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:13.154 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:13.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 10 mgrc shutdown 2023-09-30T00:22:13.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 1 -- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1690098780 msgr2=0x7f169009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:13.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1690098780 0x7f169009ac30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f16980099b0 tx=0x7f1698016000 comp rx=0 tx=0).stop 2023-09-30T00:22:13.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 1 -- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b0067a60 msgr2=0x7f16b00a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:13.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b0067a60 0x7f16b00a0ec0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f16a000c7d0 tx=0x7f16a0008040 comp rx=0 tx=0).stop 2023-09-30T00:22:13.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 1 -- 172.21.15.94:0/2054124208 shutdown_connections 2023-09-30T00:22:13.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f1690098780 0x7f169009ac30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:13.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16b0067a60 0x7f16b00a0ec0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:13.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16b00b5440 0x7f16b0154160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:13.156 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 1 --2- 172.21.15.94:0/2054124208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16b00a6070 0x7f16b00a41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:13.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 1 -- 172.21.15.94:0/2054124208 >> 172.21.15.94:0/2054124208 conn(0x7f16b006cb10 msgr2=0x7f16b00a4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:13.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 1 -- 172.21.15.94:0/2054124208 shutdown_connections 2023-09-30T00:22:13.157 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:13.155+0000 7f16b54f3700 1 -- 172.21.15.94:0/2054124208 wait complete. 2023-09-30T00:22:16.169 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:16.294 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.293+0000 7f60e359e700 1 Processor -- start 2023-09-30T00:22:16.294 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.293+0000 7f60e359e700 1 -- start start 2023-09-30T00:22:16.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e359e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40b6e50 0x7f60e40a1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:16.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e359e700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f60e40a2140 0x7f60e40a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:16.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e359e700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e40b7220 con 0x7f60e413fe20 2023-09-30T00:22:16.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e359e700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e40a2870 con 0x7f60e40a2140 2023-09-30T00:22:16.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e359e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e40a2b30 con 0x7f60e40b6e50 2023-09-30T00:22:16.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e259c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40b6e50 0x7f60e40a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:16.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e259c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40b6e50 0x7f60e40a1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51310/0 (socket says 172.21.15.94:51310) 2023-09-30T00:22:16.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e259c700 1 -- 172.21.15.94:0/3286612773 learned_addr learned my addr 172.21.15.94:0/3286612773 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:16.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e1d9b700 1 --2- 172.21.15.94:0/3286612773 >> v2:172.21.15.121:3300/0 conn(0x7f60e40a2140 0x7f60e40a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:16.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e259c700 1 -- 172.21.15.94:0/3286612773 >> v2:172.21.15.121:3300/0 conn(0x7f60e40a2140 msgr2=0x7f60e40a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:16.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e259c700 1 --2- 172.21.15.94:0/3286612773 >> v2:172.21.15.121:3300/0 conn(0x7f60e40a2140 0x7f60e40a2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:16.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e259c700 1 -- 172.21.15.94:0/3286612773 >> v1:172.21.15.121:6789/0 conn(0x7f60e413fe20 legacy=0x7f60e40a7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:16.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e259c700 1 -- 172.21.15.94:0/3286612773 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e40a2530 con 0x7f60e40b6e50 2023-09-30T00:22:16.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.294+0000 7f60e259c700 1 --2- 172.21.15.94:0/3286612773 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40b6e50 0x7f60e40a1080 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f60d40099b0 tx=0x7f60d4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80d2706c6c5e12e6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:16.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.295+0000 7f60e159a700 1 -- 172.21.15.94:0/3286612773 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d400b860 con 0x7f60e40b6e50 2023-09-30T00:22:16.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.295+0000 7f60e159a700 1 -- 172.21.15.94:0/3286612773 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d400b9c0 con 0x7f60e40b6e50 2023-09-30T00:22:16.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.295+0000 7f60e159a700 1 -- 172.21.15.94:0/3286612773 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d400bd60 con 0x7f60e40b6e50 2023-09-30T00:22:16.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.295+0000 7f60e359e700 1 -- 172.21.15.94:0/3286612773 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40b6e50 msgr2=0x7f60e40a1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:16.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.295+0000 7f60e359e700 1 --2- 172.21.15.94:0/3286612773 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40b6e50 0x7f60e40a1080 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f60d40099b0 tx=0x7f60d4009250 comp rx=0 tx=0).stop 2023-09-30T00:22:16.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.295+0000 7f60e359e700 1 -- 172.21.15.94:0/3286612773 shutdown_connections 2023-09-30T00:22:16.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.295+0000 7f60e359e700 1 --2- 172.21.15.94:0/3286612773 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40b6e50 0x7f60e40a1080 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:16.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.295+0000 7f60e359e700 1 --2- 172.21.15.94:0/3286612773 >> v2:172.21.15.121:3300/0 conn(0x7f60e40a2140 0x7f60e40a2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:16.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.295+0000 7f60e359e700 1 -- 172.21.15.94:0/3286612773 >> 172.21.15.94:0/3286612773 conn(0x7f60e40702f0 msgr2=0x7f60e40706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:16.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.295+0000 7f60e359e700 1 -- 172.21.15.94:0/3286612773 shutdown_connections 2023-09-30T00:22:16.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.295+0000 7f60e359e700 1 -- 172.21.15.94:0/3286612773 wait complete. 2023-09-30T00:22:16.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.296+0000 7f60e359e700 1 Processor -- start 2023-09-30T00:22:16.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.296+0000 7f60e359e700 1 -- start start 2023-09-30T00:22:16.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e359e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40a2140 0x7f60e40a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:16.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e359e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60e40b6e50 0x7f60e41431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:16.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60e413fe20 0x7f60e4143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:16.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e359e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e413ad30 con 0x7f60e40b6e50 2023-09-30T00:22:16.303 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e359e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4143c70 con 0x7f60e413fe20 2023-09-30T00:22:16.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e359e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e413aec0 con 0x7f60e40a2140 2023-09-30T00:22:16.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e259c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40a2140 0x7f60e40a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:16.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e259c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40a2140 0x7f60e40a6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51312/0 (socket says 172.21.15.94:51312) 2023-09-30T00:22:16.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e259c700 1 -- 172.21.15.94:0/2064414815 learned_addr learned my addr 172.21.15.94:0/2064414815 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:16.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e1d9b700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60e40b6e50 0x7f60e41431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:16.304 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e259c700 1 -- 172.21.15.94:0/2064414815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60e413fe20 msgr2=0x7f60e4143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:16.305 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e259c700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60e413fe20 0x7f60e4143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:16.305 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e259c700 1 -- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60e40b6e50 msgr2=0x7f60e41431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:16.305 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e259c700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60e40b6e50 0x7f60e41431f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:16.305 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e259c700 1 -- 172.21.15.94:0/2064414815 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d40094f0 con 0x7f60e40a2140 2023-09-30T00:22:16.305 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60e259c700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40a2140 0x7f60e40a6650 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f60e40a2c70 tx=0x7f60d4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:16.305 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.297+0000 7f60d27fc700 1 -- 172.21.15.94:0/2064414815 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d4024440 con 0x7f60e40a2140 2023-09-30T00:22:16.305 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.298+0000 7f60d27fc700 1 -- 172.21.15.94:0/2064414815 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d40245c0 con 0x7f60e40a2140 2023-09-30T00:22:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.298+0000 7f60d27fc700 1 -- 172.21.15.94:0/2064414815 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d4024980 con 0x7f60e40a2140 2023-09-30T00:22:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.298+0000 7f60e359e700 1 -- 172.21.15.94:0/2064414815 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e4144140 con 0x7f60e40a2140 2023-09-30T00:22:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.298+0000 7f60e359e700 1 -- 172.21.15.94:0/2064414815 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e40b6310 con 0x7f60e40a2140 2023-09-30T00:22:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.298+0000 7f60e359e700 1 -- 172.21.15.94:0/2064414815 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e41413d0 con 0x7f60e40a2140 2023-09-30T00:22:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.299+0000 7f60d27fc700 1 -- 172.21.15.94:0/2064414815 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f60d401a420 con 0x7f60e40a2140 2023-09-30T00:22:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.299+0000 7f60d27fc700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:16.306 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.300+0000 7f60d27fc700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.300+0000 7f60d27fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.300+0000 7f60d27fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.300+0000 7f60d27fc700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f60c4098780 0x7f60c409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.300+0000 7f60d27fc700 1 -- 172.21.15.94:0/2064414815 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f60d40c2570 con 0x7f60e40a2140 2023-09-30T00:22:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.300+0000 7f60e1d9b700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f60c4098780 0x7f60c409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:16.307 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.300+0000 7f60e1d9b700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f60c4098780 0x7f60c409ac30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f60e4141d40 tx=0x7f60d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:16.308 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.303+0000 7f60d27fc700 1 -- 172.21.15.94:0/2064414815 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f60d408fc40 con 0x7f60e40a2140 2023-09-30T00:22:16.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.529+0000 7f60e359e700 1 -- 172.21.15.94:0/2064414815 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f60e4143040 con 0x7f60e40a2140 2023-09-30T00:22:16.530 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.529+0000 7f60d27fc700 1 -- 172.21.15.94:0/2064414815 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f60d4093260 con 0x7f60e40a2140 2023-09-30T00:22:16.530 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:16.531 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:16.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.531+0000 7f60e359e700 10 mgrc shutdown 2023-09-30T00:22:16.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.531+0000 7f60e359e700 1 -- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f60c4098780 msgr2=0x7f60c409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:16.532 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.531+0000 7f60e359e700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f60c4098780 0x7f60c409ac30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f60e4141d40 tx=0x7f60d8008040 comp rx=0 tx=0).stop 2023-09-30T00:22:16.533 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.531+0000 7f60e359e700 1 -- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40a2140 msgr2=0x7f60e40a6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:16.533 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.531+0000 7f60e359e700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40a2140 0x7f60e40a6650 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f60e40a2c70 tx=0x7f60d4017000 comp rx=0 tx=0).stop 2023-09-30T00:22:16.533 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.531+0000 7f60e359e700 1 -- 172.21.15.94:0/2064414815 shutdown_connections 2023-09-30T00:22:16.533 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.531+0000 7f60e359e700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f60c4098780 0x7f60c409ac30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:16.533 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.531+0000 7f60e359e700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e40a2140 0x7f60e40a6650 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:16.533 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.531+0000 7f60e359e700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60e413fe20 0x7f60e4143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:16.533 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.531+0000 7f60e359e700 1 --2- 172.21.15.94:0/2064414815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60e40b6e50 0x7f60e41431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:16.533 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.531+0000 7f60e359e700 1 -- 172.21.15.94:0/2064414815 >> 172.21.15.94:0/2064414815 conn(0x7f60e40702f0 msgr2=0x7f60e40a6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:16.533 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.531+0000 7f60e359e700 1 -- 172.21.15.94:0/2064414815 shutdown_connections 2023-09-30T00:22:16.533 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:16.532+0000 7f60e359e700 1 -- 172.21.15.94:0/2064414815 wait complete. 2023-09-30T00:22:19.544 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.668+0000 7f47b3643700 1 Processor -- start 2023-09-30T00:22:19.669 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.669+0000 7f47b3643700 1 -- start start 2023-09-30T00:22:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.669+0000 7f47b3643700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47ac070c50 0x7f47ac0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:19.670 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.669+0000 7f47b3643700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac0b7160 0x7f47ac0a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:19.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.669+0000 7f47b3643700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac0a1310 con 0x7f47ac13fd70 2023-09-30T00:22:19.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.669+0000 7f47b3643700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ac071020 con 0x7f47ac070c50 2023-09-30T00:22:19.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.669+0000 7f47b3643700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac0b76a0 con 0x7f47ac0b7160 2023-09-30T00:22:19.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.669+0000 7f47b0bde700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac0b7160 0x7f47ac0a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:19.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.669+0000 7f47b13df700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47ac070c50 0x7f47ac0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:19.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.669+0000 7f47b1be0700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f47ac13fd70 0x7f47ac0b4dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:55798/0 (socket says 172.21.15.94:55798) 2023-09-30T00:22:19.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.669+0000 7f47b0bde700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac0b7160 0x7f47ac0a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51322/0 (socket says 172.21.15.94:51322) 2023-09-30T00:22:19.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.669+0000 7f47b1be0700 1 -- 172.21.15.94:0/2814524698 learned_addr learned my addr 172.21.15.94:0/2814524698 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:22:19.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.669+0000 7f47b13df700 1 --2- 172.21.15.94:0/2814524698 >> v2:172.21.15.94:3300/0 conn(0x7f47ac070c50 0x7f47ac0a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59784/0 (socket says 172.21.15.94:59784) 2023-09-30T00:22:19.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.670+0000 7f47b13df700 1 -- 172.21.15.94:0/2814524698 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac0b7160 msgr2=0x7f47ac0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:19.671 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.670+0000 7f47b13df700 1 --2- 172.21.15.94:0/2814524698 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac0b7160 0x7f47ac0a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:19.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.670+0000 7f47b13df700 1 -- 172.21.15.94:0/2814524698 >> v1:172.21.15.94:6789/0 conn(0x7f47ac13fd70 legacy=0x7f47ac0b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:19.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.670+0000 7f47b13df700 1 -- 172.21.15.94:0/2814524698 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac0b7940 con 0x7f47ac070c50 2023-09-30T00:22:19.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.670+0000 7f47a3fff700 1 -- 172.21.15.94:0/2814524698 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447690251 0 0) 0x7f47a4001490 con 0x7f47ac13fd70 2023-09-30T00:22:19.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.670+0000 7f47b13df700 1 --2- 172.21.15.94:0/2814524698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ac070c50 0x7f47ac0a0dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f47a80098e0 tx=0x7f47a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca455085f0760c5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:19.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.670+0000 7f47a3fff700 1 -- 172.21.15.94:0/2814524698 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a800b6d0 con 0x7f47ac070c50 2023-09-30T00:22:19.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.670+0000 7f47a3fff700 1 -- 172.21.15.94:0/2814524698 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a800b830 con 0x7f47ac070c50 2023-09-30T00:22:19.672 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.670+0000 7f47a3fff700 1 -- 172.21.15.94:0/2814524698 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a800bbd0 con 0x7f47ac070c50 2023-09-30T00:22:19.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.671+0000 7f47b3643700 1 -- 172.21.15.94:0/2814524698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ac070c50 msgr2=0x7f47ac0a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:19.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.671+0000 7f47b3643700 1 --2- 172.21.15.94:0/2814524698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ac070c50 0x7f47ac0a0dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f47a80098e0 tx=0x7f47a80091a0 comp rx=0 tx=0).stop 2023-09-30T00:22:19.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.671+0000 7f47b3643700 1 -- 172.21.15.94:0/2814524698 shutdown_connections 2023-09-30T00:22:19.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.671+0000 7f47b3643700 1 --2- 172.21.15.94:0/2814524698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ac070c50 0x7f47ac0a0dd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:19.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.671+0000 7f47b3643700 1 --2- 172.21.15.94:0/2814524698 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac0b7160 0x7f47ac0a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:19.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.671+0000 7f47b3643700 1 -- 172.21.15.94:0/2814524698 >> 172.21.15.94:0/2814524698 conn(0x7f47ac06a090 msgr2=0x7f47ac0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:19.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.671+0000 7f47b3643700 1 -- 172.21.15.94:0/2814524698 shutdown_connections 2023-09-30T00:22:19.675 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.671+0000 7f47b3643700 1 -- 172.21.15.94:0/2814524698 wait complete. 2023-09-30T00:22:19.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.672+0000 7f47b3643700 1 Processor -- start 2023-09-30T00:22:19.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.672+0000 7f47b3643700 1 -- start start 2023-09-30T00:22:19.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.672+0000 7f47b3643700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac070c50 0x7f47ac144280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:19.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.672+0000 7f47b3643700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ac0b7160 0x7f47ac13a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:19.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b3643700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47ac13fd70 0x7f47ac1388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:19.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b3643700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac139180 con 0x7f47ac0b7160 2023-09-30T00:22:19.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b3643700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac139440 con 0x7f47ac13fd70 2023-09-30T00:22:19.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b3643700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac13a610 con 0x7f47ac070c50 2023-09-30T00:22:19.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b13df700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac070c50 0x7f47ac144280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:19.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b13df700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac070c50 0x7f47ac144280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51338/0 (socket says 172.21.15.94:51338) 2023-09-30T00:22:19.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b13df700 1 -- 172.21.15.94:0/3751563625 learned_addr learned my addr 172.21.15.94:0/3751563625 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:19.676 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b0bde700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ac0b7160 0x7f47ac13a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b1be0700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47ac13fd70 0x7f47ac1388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b13df700 1 -- 172.21.15.94:0/3751563625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47ac13fd70 msgr2=0x7f47ac1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b13df700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47ac13fd70 0x7f47ac1388a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b13df700 1 -- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ac0b7160 msgr2=0x7f47ac13a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b13df700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ac0b7160 0x7f47ac13a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b13df700 1 -- 172.21.15.94:0/3751563625 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a8009420 con 0x7f47ac070c50 2023-09-30T00:22:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b1be0700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47ac13fd70 0x7f47ac1388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:22:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b0bde700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ac0b7160 0x7f47ac13a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b13df700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac070c50 0x7f47ac144280 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f47ac0a9730 tx=0x7f47a8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47a17fa700 1 -- 172.21.15.94:0/3751563625 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a800bbb0 con 0x7f47ac070c50 2023-09-30T00:22:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47a17fa700 1 -- 172.21.15.94:0/3751563625 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a8011420 con 0x7f47ac070c50 2023-09-30T00:22:19.677 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b3643700 1 -- 172.21.15.94:0/3751563625 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ac138de0 con 0x7f47ac070c50 2023-09-30T00:22:19.678 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47a17fa700 1 -- 172.21.15.94:0/3751563625 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a80117a0 con 0x7f47ac070c50 2023-09-30T00:22:19.678 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.673+0000 7f47b3643700 1 -- 172.21.15.94:0/3751563625 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ac0abdd0 con 0x7f47ac070c50 2023-09-30T00:22:19.678 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.674+0000 7f47b3643700 1 -- 172.21.15.94:0/3751563625 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ac071630 con 0x7f47ac070c50 2023-09-30T00:22:19.678 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.676+0000 7f47a17fa700 1 -- 172.21.15.94:0/3751563625 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f47a8011b50 con 0x7f47ac070c50 2023-09-30T00:22:19.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.676+0000 7f47a17fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:19.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.676+0000 7f47a17fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:19.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.676+0000 7f47a17fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:19.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.676+0000 7f47a17fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:19.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.676+0000 7f47a17fa700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4790098780 0x7f479009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:19.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.676+0000 7f47b0bde700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4790098780 0x7f479009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:19.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.676+0000 7f47a17fa700 1 -- 172.21.15.94:0/3751563625 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f47a80c1e00 con 0x7f47ac070c50 2023-09-30T00:22:19.679 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.677+0000 7f47b0bde700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4790098780 0x7f479009ac30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f47ac13b1d0 tx=0x7f479c0091a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:19.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.680+0000 7f47a17fa700 1 -- 172.21.15.94:0/3751563625 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f47a808f440 con 0x7f47ac070c50 2023-09-30T00:22:19.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.904+0000 7f47b3643700 1 -- 172.21.15.94:0/3751563625 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f47ac0a2070 con 0x7f47ac070c50 2023-09-30T00:22:19.905 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.904+0000 7f47a17fa700 1 -- 172.21.15.94:0/3751563625 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f47a8092a60 con 0x7f47ac070c50 2023-09-30T00:22:19.905 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:19.905 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:19.907 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.906+0000 7f47b3643700 10 mgrc shutdown 2023-09-30T00:22:19.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.906+0000 7f47b3643700 1 -- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4790098780 msgr2=0x7f479009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:19.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.906+0000 7f47b3643700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4790098780 0x7f479009ac30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f47ac13b1d0 tx=0x7f479c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:22:19.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.906+0000 7f47b3643700 1 -- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac070c50 msgr2=0x7f47ac144280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:19.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.906+0000 7f47b3643700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac070c50 0x7f47ac144280 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f47ac0a9730 tx=0x7f47a8017000 comp rx=0 tx=0).stop 2023-09-30T00:22:19.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.907+0000 7f47b3643700 1 -- 172.21.15.94:0/3751563625 shutdown_connections 2023-09-30T00:22:19.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.907+0000 7f47b3643700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f4790098780 0x7f479009ac30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:19.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.907+0000 7f47b3643700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47ac070c50 0x7f47ac144280 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:19.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.907+0000 7f47b3643700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47ac13fd70 0x7f47ac1388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:19.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.907+0000 7f47b3643700 1 --2- 172.21.15.94:0/3751563625 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ac0b7160 0x7f47ac13a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:19.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.907+0000 7f47b3643700 1 -- 172.21.15.94:0/3751563625 >> 172.21.15.94:0/3751563625 conn(0x7f47ac06a090 msgr2=0x7f47ac13d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:19.908 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.907+0000 7f47b3643700 1 -- 172.21.15.94:0/3751563625 shutdown_connections 2023-09-30T00:22:19.909 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:19.907+0000 7f47b3643700 1 -- 172.21.15.94:0/3751563625 wait complete. 2023-09-30T00:22:22.920 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:23.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.042+0000 7fe943a7d700 1 Processor -- start 2023-09-30T00:22:23.043 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.043+0000 7fe943a7d700 1 -- start start 2023-09-30T00:22:23.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.043+0000 7fe943a7d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe93c134790 0x7fe93c134bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:23.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.043+0000 7fe943a7d700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93c067f20 con 0x7fe93c067b50 2023-09-30T00:22:23.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.043+0000 7fe943a7d700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93c0a5180 con 0x7fe93c0b50b0 2023-09-30T00:22:23.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.043+0000 7fe943a7d700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93c141a60 con 0x7fe93c134790 2023-09-30T00:22:23.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.043+0000 7fe941819700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe93c067b50 0x7fe93c09e860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:55814/0 (socket says 172.21.15.94:55814) 2023-09-30T00:22:23.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.043+0000 7fe941819700 1 -- 172.21.15.94:0/314495858 learned_addr learned my addr 172.21.15.94:0/314495858 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:22:23.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe941018700 1 --2- 172.21.15.94:0/314495858 >> v2:172.21.15.121:3300/0 conn(0x7fe93c134790 0x7fe93c134bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:23.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092771005 0 0) 0x7fe93c067f20 con 0x7fe93c067b50 2023-09-30T00:22:23.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe928003570 con 0x7fe93c067b50 2023-09-30T00:22:23.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3602936627 0 0) 0x7fe928003570 con 0x7fe93c067b50 2023-09-30T00:22:23.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 >> v2:172.21.15.121:3300/0 conn(0x7fe93c134790 msgr2=0x7fe93c134bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:23.047 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 --2- 172.21.15.94:0/314495858 >> v2:172.21.15.121:3300/0 conn(0x7fe93c134790 0x7fe93c134bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:23.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 >> v1:172.21.15.121:6789/0 conn(0x7fe93c0b50b0 legacy=0x7fe93c0b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:23.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0b4bd0 con 0x7fe93c067b50 2023-09-30T00:22:23.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3153706020 0 0) 0x7fe9380017f0 con 0x7fe93c067b50 2023-09-30T00:22:23.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 >> v1:172.21.15.94:6789/0 conn(0x7fe93c067b50 legacy=0x7fe93c09e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:23.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 --2- 172.21.15.94:0/314495858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe928003a60 0x7fe928003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:23.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 --2- 172.21.15.94:0/314495858 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 0x7fe928004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:23.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 --2- 172.21.15.94:0/314495858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9280051c0 0x7fe92800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:23.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c141a60 con 0x7fe9280051c0 2023-09-30T00:22:23.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92800c0d0 con 0x7fe928003a60 2023-09-30T00:22:23.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92800c290 con 0x7fe928004810 2023-09-30T00:22:23.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe941018700 1 --2- 172.21.15.94:0/314495858 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 0x7fe928004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:23.048 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe941819700 1 --2- 172.21.15.94:0/314495858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe928003a60 0x7fe928003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe94201a700 1 --2- 172.21.15.94:0/314495858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9280051c0 0x7fe92800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe941018700 1 -- 172.21.15.94:0/314495858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe928003a60 msgr2=0x7fe928003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe941018700 1 --2- 172.21.15.94:0/314495858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe928003a60 0x7fe928003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe941018700 1 -- 172.21.15.94:0/314495858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9280051c0 msgr2=0x7fe92800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe941018700 1 --2- 172.21.15.94:0/314495858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9280051c0 0x7fe92800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe941018700 1 -- 172.21.15.94:0/314495858 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe928003570 con 0x7fe928004810 2023-09-30T00:22:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.044+0000 7fe94201a700 1 -- 172.21.15.94:0/314495858 reap_dead start 2023-09-30T00:22:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.045+0000 7fe941018700 1 --2- 172.21.15.94:0/314495858 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 0x7fe928004c80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe92c009c00 tx=0x7fe92c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52beecf59e737af0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.045+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c00e710 con 0x7fe928004810 2023-09-30T00:22:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.045+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c00e870 con 0x7fe928004810 2023-09-30T00:22:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.045+0000 7fe940817700 1 -- 172.21.15.94:0/314495858 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c00ec30 con 0x7fe928004810 2023-09-30T00:22:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.045+0000 7fe943a7d700 1 -- 172.21.15.94:0/314495858 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 msgr2=0x7fe928004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:23.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.045+0000 7fe943a7d700 1 --2- 172.21.15.94:0/314495858 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 0x7fe928004c80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe92c009c00 tx=0x7fe92c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:22:23.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.045+0000 7fe943a7d700 1 -- 172.21.15.94:0/314495858 shutdown_connections 2023-09-30T00:22:23.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.045+0000 7fe943a7d700 1 --2- 172.21.15.94:0/314495858 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 0x7fe928004c80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:23.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.045+0000 7fe943a7d700 1 -- 172.21.15.94:0/314495858 >> 172.21.15.94:0/314495858 conn(0x7fe93c06cb10 msgr2=0x7fe93c06cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:23.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.045+0000 7fe943a7d700 1 -- 172.21.15.94:0/314495858 shutdown_connections 2023-09-30T00:22:23.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.045+0000 7fe943a7d700 1 -- 172.21.15.94:0/314495858 wait complete. 2023-09-30T00:22:23.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.046+0000 7fe943a7d700 1 Processor -- start 2023-09-30T00:22:23.050 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.046+0000 7fe943a7d700 1 -- start start 2023-09-30T00:22:23.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe943a7d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 0x7fe93c0b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:23.052 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe943a7d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c138830 0x7fe93c136f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:23.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe943a7d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c137490 0x7fe93c14e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:23.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe943a7d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c14f430 con 0x7fe93c137490 2023-09-30T00:22:23.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe943a7d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0a2aa0 con 0x7fe93c138830 2023-09-30T00:22:23.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe943a7d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0a48b0 con 0x7fe928004810 2023-09-30T00:22:23.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe941819700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 0x7fe93c0b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:23.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe94201a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c137490 0x7fe93c14e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:23.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe941819700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 0x7fe93c0b20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:51350/0 (socket says 172.21.15.94:51350) 2023-09-30T00:22:23.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe941819700 1 -- 172.21.15.94:0/3343718374 learned_addr learned my addr 172.21.15.94:0/3343718374 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:23.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe941018700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c138830 0x7fe93c136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:23.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe941819700 1 -- 172.21.15.94:0/3343718374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c138830 msgr2=0x7fe93c136f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:23.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe941819700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c138830 0x7fe93c136f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:23.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe941819700 1 -- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c137490 msgr2=0x7fe93c14e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:23.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe941819700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c137490 0x7fe93c14e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:23.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe941819700 1 -- 172.21.15.94:0/3343718374 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0a42f0 con 0x7fe928004810 2023-09-30T00:22:23.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe94201a700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c137490 0x7fe93c14e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:23.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe941819700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 0x7fe93c0b20b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe938007bc0 tx=0x7fe93c0b4d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:23.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe931ffb700 1 -- 172.21.15.94:0/3343718374 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93800ae20 con 0x7fe928004810 2023-09-30T00:22:23.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe931ffb700 1 -- 172.21.15.94:0/3343718374 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9380133f0 con 0x7fe928004810 2023-09-30T00:22:23.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe931ffb700 1 -- 172.21.15.94:0/3343718374 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9380137b0 con 0x7fe928004810 2023-09-30T00:22:23.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.047+0000 7fe943a7d700 1 -- 172.21.15.94:0/3343718374 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c138c20 con 0x7fe928004810 2023-09-30T00:22:23.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.048+0000 7fe943a7d700 1 -- 172.21.15.94:0/3343718374 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c14eea0 con 0x7fe928004810 2023-09-30T00:22:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.049+0000 7fe943a7d700 1 -- 172.21.15.94:0/3343718374 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93c071630 con 0x7fe928004810 2023-09-30T00:22:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.049+0000 7fe931ffb700 1 -- 172.21.15.94:0/3343718374 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe938013b20 con 0x7fe928004810 2023-09-30T00:22:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.049+0000 7fe931ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.050+0000 7fe931ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.050+0000 7fe931ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.050+0000 7fe931ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.050+0000 7fe931ffb700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe920098780 0x7fe92009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.050+0000 7fe941018700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe920098780 0x7fe92009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.050+0000 7fe931ffb700 1 -- 172.21.15.94:0/3343718374 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fe9380c1160 con 0x7fe928004810 2023-09-30T00:22:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.050+0000 7fe941018700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe920098780 0x7fe92009ac30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0091e0 tx=0x7fe92c01a000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:23.056 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.054+0000 7fe931ffb700 1 -- 172.21.15.94:0/3343718374 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe93808e830 con 0x7fe928004810 2023-09-30T00:22:23.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.278+0000 7fe943a7d700 1 -- 172.21.15.94:0/3343718374 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe93c135d20 con 0x7fe928004810 2023-09-30T00:22:23.279 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.278+0000 7fe931ffb700 1 -- 172.21.15.94:0/3343718374 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe938017080 con 0x7fe928004810 2023-09-30T00:22:23.279 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:23.279 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:23.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.280+0000 7fe943a7d700 10 mgrc shutdown 2023-09-30T00:22:23.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.280+0000 7fe943a7d700 1 -- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe920098780 msgr2=0x7fe92009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:23.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.280+0000 7fe943a7d700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe920098780 0x7fe92009ac30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0091e0 tx=0x7fe92c01a000 comp rx=0 tx=0).stop 2023-09-30T00:22:23.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.280+0000 7fe943a7d700 1 -- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 msgr2=0x7fe93c0b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.280+0000 7fe943a7d700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 0x7fe93c0b20b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe938007bc0 tx=0x7fe93c0b4d60 comp rx=0 tx=0).stop 2023-09-30T00:22:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.280+0000 7fe943a7d700 1 -- 172.21.15.94:0/3343718374 shutdown_connections 2023-09-30T00:22:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.280+0000 7fe943a7d700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe920098780 0x7fe92009ac30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.280+0000 7fe943a7d700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe928004810 0x7fe93c0b20b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.280+0000 7fe943a7d700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c137490 0x7fe93c14e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.280+0000 7fe943a7d700 1 --2- 172.21.15.94:0/3343718374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c138830 0x7fe93c136f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.280+0000 7fe943a7d700 1 -- 172.21.15.94:0/3343718374 >> 172.21.15.94:0/3343718374 conn(0x7fe93c06cb10 msgr2=0x7fe93c06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.281+0000 7fe943a7d700 1 -- 172.21.15.94:0/3343718374 shutdown_connections 2023-09-30T00:22:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:23.281+0000 7fe943a7d700 1 -- 172.21.15.94:0/3343718374 wait complete. 2023-09-30T00:22:26.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:26.418 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.417+0000 7f32e5154700 1 Processor -- start 2023-09-30T00:22:26.418 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.417+0000 7f32e5154700 1 -- start start 2023-09-30T00:22:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.418+0000 7f32e5154700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e0070c50 0x7f32e00a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.418+0000 7f32e5154700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f32e00b7160 0x7f32e00a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.418+0000 7f32e5154700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e00a1310 con 0x7f32e013fd70 2023-09-30T00:22:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.418+0000 7f32e5154700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e0071020 con 0x7f32e00b7160 2023-09-30T00:22:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.418+0000 7f32e5154700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00b76a0 con 0x7f32e0070c50 2023-09-30T00:22:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.418+0000 7f32ded9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e0070c50 0x7f32e00a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.418+0000 7f32ded9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e0070c50 0x7f32e00a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:57526/0 (socket says 172.21.15.94:57526) 2023-09-30T00:22:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.418+0000 7f32ded9d700 1 -- 172.21.15.94:0/1085049085 learned_addr learned my addr 172.21.15.94:0/1085049085 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:26.421 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.418+0000 7f32d659c700 1 --2- 172.21.15.94:0/1085049085 >> v2:172.21.15.121:3300/0 conn(0x7f32e00b7160 0x7f32e00a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:26.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.418+0000 7f32ded9d700 1 -- 172.21.15.94:0/1085049085 >> v2:172.21.15.121:3300/0 conn(0x7f32e00b7160 msgr2=0x7f32e00a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:26.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32ded9d700 1 --2- 172.21.15.94:0/1085049085 >> v2:172.21.15.121:3300/0 conn(0x7f32e00b7160 0x7f32e00a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:26.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32ded9d700 1 -- 172.21.15.94:0/1085049085 >> v1:172.21.15.121:6789/0 conn(0x7f32e013fd70 legacy=0x7f32e00b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:26.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32ded9d700 1 -- 172.21.15.94:0/1085049085 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e00b7940 con 0x7f32e0070c50 2023-09-30T00:22:26.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32d659c700 1 --2- 172.21.15.94:0/1085049085 >> v2:172.21.15.121:3300/0 conn(0x7f32e00b7160 0x7f32e00a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:22:26.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32ded9d700 1 --2- 172.21.15.94:0/1085049085 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e0070c50 0x7f32e00a0dd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32d00099b0 tx=0x7f32d0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec0f7e142a02720e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:26.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32de59c700 1 -- 172.21.15.94:0/1085049085 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d000b8b0 con 0x7f32e0070c50 2023-09-30T00:22:26.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32de59c700 1 -- 172.21.15.94:0/1085049085 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d000ba10 con 0x7f32e0070c50 2023-09-30T00:22:26.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32de59c700 1 -- 172.21.15.94:0/1085049085 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d000bdb0 con 0x7f32e0070c50 2023-09-30T00:22:26.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32e5154700 1 -- 172.21.15.94:0/1085049085 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e0070c50 msgr2=0x7f32e00a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:26.422 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32e5154700 1 --2- 172.21.15.94:0/1085049085 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e0070c50 0x7f32e00a0dd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32d00099b0 tx=0x7f32d0009250 comp rx=0 tx=0).stop 2023-09-30T00:22:26.423 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32e5154700 1 -- 172.21.15.94:0/1085049085 shutdown_connections 2023-09-30T00:22:26.423 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32e5154700 1 --2- 172.21.15.94:0/1085049085 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e0070c50 0x7f32e00a0dd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:26.423 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32e5154700 1 --2- 172.21.15.94:0/1085049085 >> v2:172.21.15.121:3300/0 conn(0x7f32e00b7160 0x7f32e00a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:26.423 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32e5154700 1 -- 172.21.15.94:0/1085049085 >> 172.21.15.94:0/1085049085 conn(0x7f32e006a090 msgr2=0x7f32e00702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:26.423 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32e5154700 1 -- 172.21.15.94:0/1085049085 shutdown_connections 2023-09-30T00:22:26.423 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.419+0000 7f32e5154700 1 -- 172.21.15.94:0/1085049085 wait complete. 2023-09-30T00:22:26.423 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32e5154700 1 Processor -- start 2023-09-30T00:22:26.423 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32e5154700 1 -- start start 2023-09-30T00:22:26.425 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32e5154700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32e0070c50 0x7f32e0143b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:26.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32e5154700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32e00b7160 0x7f32e0153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:26.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32e5154700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e013fd70 0x7f32e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:26.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32e5154700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00aada0 con 0x7f32e00b7160 2023-09-30T00:22:26.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32e5154700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e0144c10 con 0x7f32e0070c50 2023-09-30T00:22:26.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32e5154700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e0154540 con 0x7f32e013fd70 2023-09-30T00:22:26.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32df59e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e013fd70 0x7f32e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:26.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32d659c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32e00b7160 0x7f32e0153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:26.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32df59e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e013fd70 0x7f32e00a6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:57532/0 (socket says 172.21.15.94:57532) 2023-09-30T00:22:26.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32df59e700 1 -- 172.21.15.94:0/4155851292 learned_addr learned my addr 172.21.15.94:0/4155851292 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:26.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.421+0000 7f32ded9d700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32e0070c50 0x7f32e0143b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:26.426 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.422+0000 7f32df59e700 1 -- 172.21.15.94:0/4155851292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32e0070c50 msgr2=0x7f32e0143b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:26.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.422+0000 7f32df59e700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32e0070c50 0x7f32e0143b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:26.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.422+0000 7f32df59e700 1 -- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32e00b7160 msgr2=0x7f32e0153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:26.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.422+0000 7f32df59e700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32e00b7160 0x7f32e0153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:26.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.422+0000 7f32df59e700 1 -- 172.21.15.94:0/4155851292 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d00094f0 con 0x7f32e013fd70 2023-09-30T00:22:26.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.422+0000 7f32df59e700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e013fd70 0x7f32e00a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f32d800bc60 tx=0x7f32d8009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:26.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.422+0000 7f32d77fe700 1 -- 172.21.15.94:0/4155851292 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d800c840 con 0x7f32e013fd70 2023-09-30T00:22:26.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.422+0000 7f32d77fe700 1 -- 172.21.15.94:0/4155851292 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d800c9a0 con 0x7f32e013fd70 2023-09-30T00:22:26.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.422+0000 7f32e5154700 1 -- 172.21.15.94:0/4155851292 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e0154800 con 0x7f32e013fd70 2023-09-30T00:22:26.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.422+0000 7f32d77fe700 1 -- 172.21.15.94:0/4155851292 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d800cda0 con 0x7f32e013fd70 2023-09-30T00:22:26.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.422+0000 7f32e5154700 1 -- 172.21.15.94:0/4155851292 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e00ab9f0 con 0x7f32e013fd70 2023-09-30T00:22:26.427 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.423+0000 7f32e5154700 1 -- 172.21.15.94:0/4155851292 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e013e3f0 con 0x7f32e013fd70 2023-09-30T00:22:26.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.424+0000 7f32d77fe700 1 -- 172.21.15.94:0/4155851292 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f32d8014620 con 0x7f32e013fd70 2023-09-30T00:22:26.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.424+0000 7f32d77fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:26.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.424+0000 7f32d77fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:26.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.424+0000 7f32d77fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:26.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.424+0000 7f32d77fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:26.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.424+0000 7f32d77fe700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f32c00987b0 0x7f32c009ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:26.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.424+0000 7f32ded9d700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f32c00987b0 0x7f32c009ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:26.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.424+0000 7f32d77fe700 1 -- 172.21.15.94:0/4155851292 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f32d80c0440 con 0x7f32e013fd70 2023-09-30T00:22:26.428 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.425+0000 7f32ded9d700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f32c00987b0 0x7f32c009ac60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f32d00099b0 tx=0x7f32d0016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:26.429 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.428+0000 7f32d77fe700 1 -- 172.21.15.94:0/4155851292 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f32d808da80 con 0x7f32e013fd70 2023-09-30T00:22:26.652 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.652+0000 7f32e5154700 1 -- 172.21.15.94:0/4155851292 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f32e013e110 con 0x7f32e013fd70 2023-09-30T00:22:26.653 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.652+0000 7f32d77fe700 1 -- 172.21.15.94:0/4155851292 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f32d80910a0 con 0x7f32e013fd70 2023-09-30T00:22:26.653 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:26.653 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:26.655 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 10 mgrc shutdown 2023-09-30T00:22:26.655 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 1 -- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f32c00987b0 msgr2=0x7f32c009ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:26.655 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f32c00987b0 0x7f32c009ac60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f32d00099b0 tx=0x7f32d0016000 comp rx=0 tx=0).stop 2023-09-30T00:22:26.655 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 1 -- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e013fd70 msgr2=0x7f32e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:26.655 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e013fd70 0x7f32e00a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f32d800bc60 tx=0x7f32d8009820 comp rx=0 tx=0).stop 2023-09-30T00:22:26.656 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 1 -- 172.21.15.94:0/4155851292 shutdown_connections 2023-09-30T00:22:26.656 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f32c00987b0 0x7f32c009ac60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:26.656 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f32e013fd70 0x7f32e00a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:26.656 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32e0070c50 0x7f32e0143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:26.656 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 1 --2- 172.21.15.94:0/4155851292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32e00b7160 0x7f32e0153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:26.656 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 1 -- 172.21.15.94:0/4155851292 >> 172.21.15.94:0/4155851292 conn(0x7f32e006a090 msgr2=0x7f32e013d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:26.656 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 1 -- 172.21.15.94:0/4155851292 shutdown_connections 2023-09-30T00:22:26.656 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:26.654+0000 7f32e5154700 1 -- 172.21.15.94:0/4155851292 wait complete. 2023-09-30T00:22:29.667 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:29.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.791+0000 7ffa0b46b700 1 Processor -- start 2023-09-30T00:22:29.792 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.792+0000 7ffa0b46b700 1 -- start start 2023-09-30T00:22:29.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.792+0000 7ffa0b46b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ffa04070c50 0x7ffa040a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:29.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.792+0000 7ffa0b46b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffa0413fd70 0x7ffa04140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:29.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.792+0000 7ffa0b46b700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa040a1310 con 0x7ffa040b7160 2023-09-30T00:22:29.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.792+0000 7ffa0b46b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa04071020 con 0x7ffa0413fd70 2023-09-30T00:22:29.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.792+0000 7ffa0b46b700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa040b76a0 con 0x7ffa04070c50 2023-09-30T00:22:29.795 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.792+0000 7ffa09a08700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffa0413fd70 0x7ffa04140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:29.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.793+0000 7ffa09a08700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffa0413fd70 0x7ffa04140180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56732/0 (socket says 172.21.15.94:56732) 2023-09-30T00:22:29.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.793+0000 7ffa09a08700 1 -- 172.21.15.94:0/2724457503 learned_addr learned my addr 172.21.15.94:0/2724457503 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:29.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.793+0000 7ffa09207700 1 --2- 172.21.15.94:0/2724457503 >> v2:172.21.15.121:3300/0 conn(0x7ffa04070c50 0x7ffa040a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:29.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.793+0000 7ffa09a08700 1 -- 172.21.15.94:0/2724457503 >> v2:172.21.15.121:3300/0 conn(0x7ffa04070c50 msgr2=0x7ffa040a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:29.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.793+0000 7ffa09a08700 1 --2- 172.21.15.94:0/2724457503 >> v2:172.21.15.121:3300/0 conn(0x7ffa04070c50 0x7ffa040a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:29.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.793+0000 7ffa09a08700 1 -- 172.21.15.94:0/2724457503 >> v1:172.21.15.121:6789/0 conn(0x7ffa040b7160 legacy=0x7ffa040a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:29.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.793+0000 7ffa09a08700 1 -- 172.21.15.94:0/2724457503 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa040b7940 con 0x7ffa0413fd70 2023-09-30T00:22:29.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.793+0000 7ffa09a08700 1 --2- 172.21.15.94:0/2724457503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa0413fd70 0x7ffa04140180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ffa040a9760 tx=0x7ff9fc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f891e72c037c27d3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:29.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.793+0000 7ff9fbfff700 1 -- 172.21.15.94:0/2724457503 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc00e070 con 0x7ffa0413fd70 2023-09-30T00:22:29.796 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.793+0000 7ff9fbfff700 1 -- 172.21.15.94:0/2724457503 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9fc00bab0 con 0x7ffa0413fd70 2023-09-30T00:22:29.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.793+0000 7ff9fbfff700 1 -- 172.21.15.94:0/2724457503 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc0092e0 con 0x7ffa0413fd70 2023-09-30T00:22:29.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.794+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2724457503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa0413fd70 msgr2=0x7ffa04140180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:29.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.794+0000 7ffa0b46b700 1 --2- 172.21.15.94:0/2724457503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa0413fd70 0x7ffa04140180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ffa040a9760 tx=0x7ff9fc009840 comp rx=0 tx=0).stop 2023-09-30T00:22:29.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.794+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2724457503 shutdown_connections 2023-09-30T00:22:29.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.794+0000 7ffa0b46b700 1 --2- 172.21.15.94:0/2724457503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa0413fd70 0x7ffa04140180 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:29.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.794+0000 7ffa0b46b700 1 --2- 172.21.15.94:0/2724457503 >> v2:172.21.15.121:3300/0 conn(0x7ffa04070c50 0x7ffa040a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:29.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.794+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2724457503 >> 172.21.15.94:0/2724457503 conn(0x7ffa0406a090 msgr2=0x7ffa040702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:29.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.794+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2724457503 shutdown_connections 2023-09-30T00:22:29.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.794+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2724457503 wait complete. 2023-09-30T00:22:29.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.795+0000 7ffa0b46b700 1 Processor -- start 2023-09-30T00:22:29.797 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.795+0000 7ffa0b46b700 1 -- start start 2023-09-30T00:22:29.800 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.795+0000 7ffa0b46b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa04070c50 0x7ffa0413c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:29.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.795+0000 7ffa0b46b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa040b7160 0x7ffa040a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:29.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.795+0000 7ffa0b46b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa0413fd70 0x7ffa04137910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:29.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.795+0000 7ffa0b46b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa04144740 con 0x7ffa04070c50 2023-09-30T00:22:29.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.795+0000 7ffa0b46b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa040a5d80 con 0x7ffa0413fd70 2023-09-30T00:22:29.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.795+0000 7ffa0b46b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa04139c70 con 0x7ffa040b7160 2023-09-30T00:22:29.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.795+0000 7ffa09207700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa04070c50 0x7ffa0413c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:29.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.795+0000 7ffa09207700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa04070c50 0x7ffa0413c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56736/0 (socket says 172.21.15.94:56736) 2023-09-30T00:22:29.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.795+0000 7ffa08a06700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa040b7160 0x7ffa040a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:29.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ffa09207700 1 -- 172.21.15.94:0/2349722364 learned_addr learned my addr 172.21.15.94:0/2349722364 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:29.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ffa09a08700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa0413fd70 0x7ffa04137910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:29.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ffa09207700 1 -- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa040b7160 msgr2=0x7ffa040a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:29.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ffa09207700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa040b7160 0x7ffa040a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:29.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ffa09207700 1 -- 172.21.15.94:0/2349722364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa0413fd70 msgr2=0x7ffa04137910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:29.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ffa09207700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa0413fd70 0x7ffa04137910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:29.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ffa09207700 1 -- 172.21.15.94:0/2349722364 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9fc009ae0 con 0x7ffa04070c50 2023-09-30T00:22:29.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ffa08a06700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa040b7160 0x7ffa040a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ffa09207700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa04070c50 0x7ffa0413c340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000ec80 tx=0x7ffa000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ff9f97fa700 1 -- 172.21.15.94:0/2349722364 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0000c850 con 0x7ffa04070c50 2023-09-30T00:22:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ff9f97fa700 1 -- 172.21.15.94:0/2349722364 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0000c9b0 con 0x7ffa04070c50 2023-09-30T00:22:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2349722364 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa040a4c00 con 0x7ffa04070c50 2023-09-30T00:22:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ff9f97fa700 1 -- 172.21.15.94:0/2349722364 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0000cd50 con 0x7ffa04070c50 2023-09-30T00:22:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.796+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2349722364 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0413c900 con 0x7ffa04070c50 2023-09-30T00:22:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.797+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2349722364 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa04071630 con 0x7ffa04070c50 2023-09-30T00:22:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.798+0000 7ff9f97fa700 1 -- 172.21.15.94:0/2349722364 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7ffa00013680 con 0x7ffa04070c50 2023-09-30T00:22:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.798+0000 7ff9f97fa700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:29.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.799+0000 7ff9f97fa700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.799+0000 7ff9f97fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.799+0000 7ff9f97fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.799+0000 7ff9f97fa700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff9e8098780 0x7ff9e809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.799+0000 7ffa08a06700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff9e8098780 0x7ff9e809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.799+0000 7ff9f97fa700 1 -- 172.21.15.94:0/2349722364 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7ffa000c0f50 con 0x7ffa04070c50 2023-09-30T00:22:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.799+0000 7ffa08a06700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff9e8098780 0x7ff9e809ac30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffa0413f9c0 tx=0x7ff9f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:29.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:29.802+0000 7ff9f97fa700 1 -- 172.21.15.94:0/2349722364 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffa0008e4f0 con 0x7ffa04070c50 2023-09-30T00:22:30.025 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.023+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2349722364 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ffa040a2070 con 0x7ffa04070c50 2023-09-30T00:22:30.025 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.024+0000 7ff9f97fa700 1 -- 172.21.15.94:0/2349722364 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ffa00021100 con 0x7ffa04070c50 2023-09-30T00:22:30.025 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:30.025 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:30.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 10 mgrc shutdown 2023-09-30T00:22:30.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff9e8098780 msgr2=0x7ff9e809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:30.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff9e8098780 0x7ff9e809ac30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffa0413f9c0 tx=0x7ff9f4009040 comp rx=0 tx=0).stop 2023-09-30T00:22:30.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa04070c50 msgr2=0x7ffa0413c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:30.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa04070c50 0x7ffa0413c340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000ec80 tx=0x7ffa000091a0 comp rx=0 tx=0).stop 2023-09-30T00:22:30.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2349722364 shutdown_connections 2023-09-30T00:22:30.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7ff9e8098780 0x7ff9e809ac30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:30.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa0413fd70 0x7ffa04137910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:30.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa04070c50 0x7ffa0413c340 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:30.028 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 1 --2- 172.21.15.94:0/2349722364 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa040b7160 0x7ffa040a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:30.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2349722364 >> 172.21.15.94:0/2349722364 conn(0x7ffa0406a090 msgr2=0x7ffa04138e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:30.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2349722364 shutdown_connections 2023-09-30T00:22:30.029 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:30.026+0000 7ffa0b46b700 1 -- 172.21.15.94:0/2349722364 wait complete. 2023-09-30T00:22:33.040 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:33.164 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.163+0000 7f557a50b700 1 Processor -- start 2023-09-30T00:22:33.165 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.163+0000 7f557a50b700 1 -- start start 2023-09-30T00:22:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f557a50b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55740b2cb0 0x7f5574143db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f557a50b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f55741442f0 0x7f5574136450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f557a50b700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557406ae70 con 0x7f557406aaa0 2023-09-30T00:22:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f557a50b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55740a6390 con 0x7f55740b2cb0 2023-09-30T00:22:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f557a50b700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5574144790 con 0x7f55741442f0 2023-09-30T00:22:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f55737fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55740b2cb0 0x7f5574143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f55737fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55740b2cb0 0x7f5574143db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56752/0 (socket says 172.21.15.94:56752) 2023-09-30T00:22:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f55737fe700 1 -- 172.21.15.94:0/4078266171 learned_addr learned my addr 172.21.15.94:0/4078266171 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f5573fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f557406aaa0 0x7f55740b25a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42208/0 (socket says 172.21.15.94:42208) 2023-09-30T00:22:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f55737fe700 1 -- 172.21.15.94:0/4078266171 >> v2:172.21.15.121:3300/0 conn(0x7f55741442f0 msgr2=0x7f5574136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f55737fe700 1 --2- 172.21.15.94:0/4078266171 >> v2:172.21.15.121:3300/0 conn(0x7f55741442f0 0x7f5574136450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f55737fe700 1 -- 172.21.15.94:0/4078266171 >> v1:172.21.15.94:6789/0 conn(0x7f557406aaa0 legacy=0x7f55740b25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f5572ffd700 1 -- 172.21.15.94:0/4078266171 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571822831 0 0) 0x7f557406ae70 con 0x7f557406aaa0 2023-09-30T00:22:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f55737fe700 1 -- 172.21.15.94:0/4078266171 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5574144a40 con 0x7f55740b2cb0 2023-09-30T00:22:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f55737fe700 1 --2- 172.21.15.94:0/4078266171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55740b2cb0 0x7f5574143db0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f557406b550 tx=0x7f5564009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef0dda9330f377b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f5572ffd700 1 -- 172.21.15.94:0/4078266171 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556400b860 con 0x7f55740b2cb0 2023-09-30T00:22:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f5572ffd700 1 -- 172.21.15.94:0/4078266171 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556400b9c0 con 0x7f55740b2cb0 2023-09-30T00:22:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.164+0000 7f5572ffd700 1 -- 172.21.15.94:0/4078266171 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556400bd60 con 0x7f55740b2cb0 2023-09-30T00:22:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.165+0000 7f557a50b700 1 -- 172.21.15.94:0/4078266171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55740b2cb0 msgr2=0x7f5574143db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.165+0000 7f557a50b700 1 --2- 172.21.15.94:0/4078266171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55740b2cb0 0x7f5574143db0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f557406b550 tx=0x7f5564009250 comp rx=0 tx=0).stop 2023-09-30T00:22:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.165+0000 7f557a50b700 1 -- 172.21.15.94:0/4078266171 shutdown_connections 2023-09-30T00:22:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.165+0000 7f557a50b700 1 --2- 172.21.15.94:0/4078266171 >> v2:172.21.15.121:3300/0 conn(0x7f55741442f0 0x7f5574136450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.165+0000 7f557a50b700 1 --2- 172.21.15.94:0/4078266171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55740b2cb0 0x7f5574143db0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.165+0000 7f557a50b700 1 -- 172.21.15.94:0/4078266171 >> 172.21.15.94:0/4078266171 conn(0x7f5574068730 msgr2=0x7f5574068b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.165+0000 7f557a50b700 1 -- 172.21.15.94:0/4078266171 shutdown_connections 2023-09-30T00:22:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.165+0000 7f557a50b700 1 -- 172.21.15.94:0/4078266171 wait complete. 2023-09-30T00:22:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.166+0000 7f557a50b700 1 Processor -- start 2023-09-30T00:22:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.166+0000 7f557a50b700 1 -- start start 2023-09-30T00:22:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f557a50b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557406aaa0 0x7f557409f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f557a50b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55740b2cb0 0x7f557414fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f557a50b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55741442f0 0x7f5574150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f557a50b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5574150930 con 0x7f557406aaa0 2023-09-30T00:22:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f557a50b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5574150bd0 con 0x7f55740b2cb0 2023-09-30T00:22:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f557a50b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5574150e70 con 0x7f55741442f0 2023-09-30T00:22:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f5573fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557406aaa0 0x7f557409f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f5573fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557406aaa0 0x7f557409f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56756/0 (socket says 172.21.15.94:56756) 2023-09-30T00:22:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f5573fff700 1 -- 172.21.15.94:0/1769313777 learned_addr learned my addr 172.21.15.94:0/1769313777 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f5578aa8700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55741442f0 0x7f5574150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f55737fe700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55740b2cb0 0x7f557414fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f5573fff700 1 -- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55741442f0 msgr2=0x7f5574150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f5573fff700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55741442f0 0x7f5574150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f5573fff700 1 -- 172.21.15.94:0/1769313777 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55740b2cb0 msgr2=0x7f557414fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f5573fff700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55740b2cb0 0x7f557414fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f5573fff700 1 -- 172.21.15.94:0/1769313777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55640094f0 con 0x7f557406aaa0 2023-09-30T00:22:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f5578aa8700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55741442f0 0x7f5574150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f5573fff700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557406aaa0 0x7f557409f530 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f55680011e0 tx=0x7f5568007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f555bfff700 1 -- 172.21.15.94:0/1769313777 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568005a70 con 0x7f557406aaa0 2023-09-30T00:22:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f555bfff700 1 -- 172.21.15.94:0/1769313777 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5568005bd0 con 0x7f557406aaa0 2023-09-30T00:22:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.167+0000 7f557a50b700 1 -- 172.21.15.94:0/1769313777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f557409f090 con 0x7f557406aaa0 2023-09-30T00:22:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.168+0000 7f555bfff700 1 -- 172.21.15.94:0/1769313777 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55680070d0 con 0x7f557406aaa0 2023-09-30T00:22:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.168+0000 7f557a50b700 1 -- 172.21.15.94:0/1769313777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f557409e760 con 0x7f557406aaa0 2023-09-30T00:22:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.168+0000 7f557a50b700 1 -- 172.21.15.94:0/1769313777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5574136fb0 con 0x7f557406aaa0 2023-09-30T00:22:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.169+0000 7f555bfff700 1 -- 172.21.15.94:0/1769313777 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f5568007480 con 0x7f557406aaa0 2023-09-30T00:22:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.169+0000 7f555bfff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.170+0000 7f555bfff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.170+0000 7f555bfff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.170+0000 7f555bfff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.170+0000 7f555bfff700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5560098780 0x7f556009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.170+0000 7f55737fe700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5560098780 0x7f556009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.170+0000 7f555bfff700 1 -- 172.21.15.94:0/1769313777 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f55680bf8b0 con 0x7f557406aaa0 2023-09-30T00:22:33.175 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.170+0000 7f55737fe700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5560098780 0x7f556009ac30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f557406b550 tx=0x7f5564017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:33.175 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.174+0000 7f555bfff700 1 -- 172.21.15.94:0/1769313777 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f556804b5a0 con 0x7f557406aaa0 2023-09-30T00:22:33.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.397+0000 7f557a50b700 1 -- 172.21.15.94:0/1769313777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f55740a0540 con 0x7f557406aaa0 2023-09-30T00:22:33.399 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.398+0000 7f555bfff700 1 -- 172.21.15.94:0/1769313777 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5568017080 con 0x7f557406aaa0 2023-09-30T00:22:33.399 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:33.399 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:33.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.399+0000 7f557a50b700 10 mgrc shutdown 2023-09-30T00:22:33.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.400+0000 7f557a50b700 1 -- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5560098780 msgr2=0x7f556009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:33.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.400+0000 7f557a50b700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5560098780 0x7f556009ac30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f557406b550 tx=0x7f5564017000 comp rx=0 tx=0).stop 2023-09-30T00:22:33.401 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.400+0000 7f557a50b700 1 -- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557406aaa0 msgr2=0x7f557409f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:33.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.400+0000 7f557a50b700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557406aaa0 0x7f557409f530 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f55680011e0 tx=0x7f5568007e10 comp rx=0 tx=0).stop 2023-09-30T00:22:33.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.400+0000 7f557a50b700 1 -- 172.21.15.94:0/1769313777 shutdown_connections 2023-09-30T00:22:33.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.400+0000 7f557a50b700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f5560098780 0x7f556009ac30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:33.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.400+0000 7f557a50b700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55741442f0 0x7f5574150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:33.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.400+0000 7f557a50b700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f557406aaa0 0x7f557409f530 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:33.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.400+0000 7f557a50b700 1 --2- 172.21.15.94:0/1769313777 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55740b2cb0 0x7f557414fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:33.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.400+0000 7f557a50b700 1 -- 172.21.15.94:0/1769313777 >> 172.21.15.94:0/1769313777 conn(0x7f5574068730 msgr2=0x7f5574068b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:33.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.400+0000 7f557a50b700 1 -- 172.21.15.94:0/1769313777 shutdown_connections 2023-09-30T00:22:33.402 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:33.400+0000 7f557a50b700 1 -- 172.21.15.94:0/1769313777 wait complete. 2023-09-30T00:22:36.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:36.539 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.537+0000 7f48ee185700 1 Processor -- start 2023-09-30T00:22:36.539 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.537+0000 7f48ee185700 1 -- start start 2023-09-30T00:22:36.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48ee185700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f48e8070c50 0x7f48e80a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:36.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48ee185700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 0x7f48e80a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:36.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48ee185700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48e80a1310 con 0x7f48e813fd70 2023-09-30T00:22:36.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48ee185700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48e8071020 con 0x7f48e8070c50 2023-09-30T00:22:36.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48ee185700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e80b76a0 con 0x7f48e80b7160 2023-09-30T00:22:36.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e6ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 0x7f48e80a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:36.541 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e77fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f48e8070c50 0x7f48e80a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e77fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f48e8070c50 0x7f48e80a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52630/0 (socket says 172.21.15.94:52630) 2023-09-30T00:22:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e6ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 0x7f48e80a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:36214/0 (socket says 172.21.15.94:36214) 2023-09-30T00:22:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e7fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f48e813fd70 0x7f48e80b4dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40016/0 (socket says 172.21.15.94:40016) 2023-09-30T00:22:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e77fe700 1 -- 172.21.15.94:0/1330042336 learned_addr learned my addr 172.21.15.94:0/1330042336 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e77fe700 1 -- 172.21.15.94:0/1330042336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 msgr2=0x7f48e80a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e77fe700 1 --2- 172.21.15.94:0/1330042336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 0x7f48e80a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e77fe700 1 -- 172.21.15.94:0/1330042336 >> v1:172.21.15.94:6789/0 conn(0x7f48e813fd70 legacy=0x7f48e80b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e77fe700 1 -- 172.21.15.94:0/1330042336 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e80b7940 con 0x7f48e8070c50 2023-09-30T00:22:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e67fc700 1 -- 172.21.15.94:0/1330042336 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922666074 0 0) 0x7f48e0001110 con 0x7f48e813fd70 2023-09-30T00:22:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e6ffd700 1 --2- 172.21.15.94:0/1330042336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 0x7f48e80a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.538+0000 7f48e77fe700 1 --2- 172.21.15.94:0/1330042336 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e8070c50 0x7f48e80a0dd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f48d8009f20 tx=0x7f48d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310506e43dfba122 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.539+0000 7f48e67fc700 1 -- 172.21.15.94:0/1330042336 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d800e070 con 0x7f48e8070c50 2023-09-30T00:22:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.539+0000 7f48e67fc700 1 -- 172.21.15.94:0/1330042336 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d800ba60 con 0x7f48e8070c50 2023-09-30T00:22:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.539+0000 7f48e67fc700 1 -- 172.21.15.94:0/1330042336 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d80092e0 con 0x7f48e8070c50 2023-09-30T00:22:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.539+0000 7f48ee185700 1 -- 172.21.15.94:0/1330042336 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e8070c50 msgr2=0x7f48e80a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.539+0000 7f48ee185700 1 --2- 172.21.15.94:0/1330042336 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e8070c50 0x7f48e80a0dd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f48d8009f20 tx=0x7f48d8009840 comp rx=0 tx=0).stop 2023-09-30T00:22:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.539+0000 7f48ee185700 1 -- 172.21.15.94:0/1330042336 shutdown_connections 2023-09-30T00:22:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.539+0000 7f48ee185700 1 --2- 172.21.15.94:0/1330042336 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e8070c50 0x7f48e80a0dd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.539+0000 7f48ee185700 1 --2- 172.21.15.94:0/1330042336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 0x7f48e80a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.539+0000 7f48ee185700 1 -- 172.21.15.94:0/1330042336 >> 172.21.15.94:0/1330042336 conn(0x7f48e806a090 msgr2=0x7f48e80702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.539+0000 7f48ee185700 1 -- 172.21.15.94:0/1330042336 shutdown_connections 2023-09-30T00:22:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.539+0000 7f48ee185700 1 -- 172.21.15.94:0/1330042336 wait complete. 2023-09-30T00:22:36.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.540+0000 7f48ee185700 1 Processor -- start 2023-09-30T00:22:36.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.540+0000 7f48ee185700 1 -- start start 2023-09-30T00:22:36.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48ee185700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48e8070c50 0x7f48e813c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:36.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48ee185700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 0x7f48e80a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:36.546 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48ee185700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e813fd70 0x7f48e8150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48ee185700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e80ad840 con 0x7f48e813fd70 2023-09-30T00:22:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48ee185700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e8151110 con 0x7f48e8070c50 2023-09-30T00:22:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48ee185700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e81513d0 con 0x7f48e80b7160 2023-09-30T00:22:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48e6ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 0x7f48e80a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48e7fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e813fd70 0x7f48e8150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48e6ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 0x7f48e80a6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:36230/0 (socket says 172.21.15.94:36230) 2023-09-30T00:22:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48e6ffd700 1 -- 172.21.15.94:0/2824164341 learned_addr learned my addr 172.21.15.94:0/2824164341 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48e7fff700 1 -- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 msgr2=0x7f48e80a6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48e7fff700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 0x7f48e80a6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48e7fff700 1 -- 172.21.15.94:0/2824164341 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48e8070c50 msgr2=0x7f48e813c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:36.547 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48e7fff700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48e8070c50 0x7f48e813c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48e7fff700 1 -- 172.21.15.94:0/2824164341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc009420 con 0x7f48e813fd70 2023-09-30T00:22:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48e6ffd700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 0x7f48e80a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48e7fff700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e813fd70 0x7f48e8150320 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f48e000c470 tx=0x7f48e0007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48cf7fe700 1 -- 172.21.15.94:0/2824164341 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e0005a70 con 0x7f48e813fd70 2023-09-30T00:22:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48cf7fe700 1 -- 172.21.15.94:0/2824164341 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e0005bd0 con 0x7f48e813fd70 2023-09-30T00:22:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48ee185700 1 -- 172.21.15.94:0/2824164341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d8009ae0 con 0x7f48e813fd70 2023-09-30T00:22:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.541+0000 7f48cf7fe700 1 -- 172.21.15.94:0/2824164341 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e00070d0 con 0x7f48e813fd70 2023-09-30T00:22:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.542+0000 7f48ee185700 1 -- 172.21.15.94:0/2824164341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e813dbd0 con 0x7f48e813fd70 2023-09-30T00:22:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.542+0000 7f48ee185700 1 -- 172.21.15.94:0/2824164341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e813e4c0 con 0x7f48e813fd70 2023-09-30T00:22:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.543+0000 7f48cf7fe700 1 -- 172.21.15.94:0/2824164341 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f48e0007480 con 0x7f48e813fd70 2023-09-30T00:22:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.543+0000 7f48cf7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.544+0000 7f48cf7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.544+0000 7f48cf7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.544+0000 7f48cf7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.544+0000 7f48cf7fe700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f48d4098780 0x7f48d409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.544+0000 7f48e77fe700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f48d4098780 0x7f48d409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.544+0000 7f48cf7fe700 1 -- 172.21.15.94:0/2824164341 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f48e00bf6f0 con 0x7f48e813fd70 2023-09-30T00:22:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.544+0000 7f48e77fe700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f48d4098780 0x7f48d409ac30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f48d8009f20 tx=0x7f48d8018000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.548+0000 7f48cf7fe700 1 -- 172.21.15.94:0/2824164341 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f48e008cd30 con 0x7f48e813fd70 2023-09-30T00:22:36.772 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.771+0000 7f48ee185700 1 -- 172.21.15.94:0/2824164341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f48e80a1310 con 0x7f48e813fd70 2023-09-30T00:22:36.773 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.771+0000 7f48cf7fe700 1 -- 172.21.15.94:0/2824164341 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f48e80a1310 con 0x7f48e813fd70 2023-09-30T00:22:36.773 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:36.773 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:36.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 10 mgrc shutdown 2023-09-30T00:22:36.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 1 -- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f48d4098780 msgr2=0x7f48d409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:36.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f48d4098780 0x7f48d409ac30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f48d8009f20 tx=0x7f48d8018000 comp rx=0 tx=0).stop 2023-09-30T00:22:36.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 1 -- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e813fd70 msgr2=0x7f48e8150320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:36.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e813fd70 0x7f48e8150320 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f48e000c470 tx=0x7f48e0007ed0 comp rx=0 tx=0).stop 2023-09-30T00:22:36.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 1 -- 172.21.15.94:0/2824164341 shutdown_connections 2023-09-30T00:22:36.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f48d4098780 0x7f48d409ac30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:36.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e813fd70 0x7f48e8150320 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:36.775 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48e8070c50 0x7f48e813c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:36.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 1 --2- 172.21.15.94:0/2824164341 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48e80b7160 0x7f48e80a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:36.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 1 -- 172.21.15.94:0/2824164341 >> 172.21.15.94:0/2824164341 conn(0x7f48e806a090 msgr2=0x7f48e80b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:36.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 1 -- 172.21.15.94:0/2824164341 shutdown_connections 2023-09-30T00:22:36.776 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:36.773+0000 7f48ee185700 1 -- 172.21.15.94:0/2824164341 wait complete. 2023-09-30T00:22:39.646 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:22:39.649 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:22:39.787 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:39.912 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.910+0000 7fc371a57700 1 Processor -- start 2023-09-30T00:22:39.912 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.910+0000 7fc371a57700 1 -- start start 2023-09-30T00:22:39.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.911+0000 7fc371a57700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c067e30 0x7fc36c0a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:39.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.911+0000 7fc371a57700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36c068200 con 0x7fc36c0a94d0 2023-09-30T00:22:39.914 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.911+0000 7fc371a57700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36c0a24a0 con 0x7fc36c0a9ea0 2023-09-30T00:22:39.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.911+0000 7fc371a57700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c0a22b0 con 0x7fc36c067e30 2023-09-30T00:22:39.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.911+0000 7fc36affd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c067e30 0x7fc36c0a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:39.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.911+0000 7fc36affd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c067e30 0x7fc36c0a1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:36242/0 (socket says 172.21.15.94:36242) 2023-09-30T00:22:39.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.911+0000 7fc36b7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc36c0a94d0 0x7fc36c14b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40028/0 (socket says 172.21.15.94:40028) 2023-09-30T00:22:39.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.911+0000 7fc36affd700 1 -- 172.21.15.94:0/145175936 learned_addr learned my addr 172.21.15.94:0/145175936 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:39.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.911+0000 7fc36affd700 1 -- 172.21.15.94:0/145175936 >> v1:172.21.15.121:6789/0 conn(0x7fc36c0a9ea0 legacy=0x7fc36c13fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:39.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.911+0000 7fc36affd700 1 -- 172.21.15.94:0/145175936 >> v1:172.21.15.94:6789/0 conn(0x7fc36c0a94d0 legacy=0x7fc36c14b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:39.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.911+0000 7fc36affd700 1 -- 172.21.15.94:0/145175936 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c0a2740 con 0x7fc36c067e30 2023-09-30T00:22:39.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.911+0000 7fc36a7fc700 1 -- 172.21.15.94:0/145175936 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862531706 0 0) 0x7fc36c068200 con 0x7fc36c0a94d0 2023-09-30T00:22:39.915 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.912+0000 7fc36affd700 1 --2- 172.21.15.94:0/145175936 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c067e30 0x7fc36c0a1d70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0099e0 tx=0x7fc35c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37bf2200eba713c5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:39.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.912+0000 7fc36a7fc700 1 -- 172.21.15.94:0/145175936 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c00b860 con 0x7fc36c067e30 2023-09-30T00:22:39.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.912+0000 7fc36a7fc700 1 -- 172.21.15.94:0/145175936 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c00b9c0 con 0x7fc36c067e30 2023-09-30T00:22:39.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.912+0000 7fc36a7fc700 1 -- 172.21.15.94:0/145175936 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c00bd60 con 0x7fc36c067e30 2023-09-30T00:22:39.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.912+0000 7fc371a57700 1 -- 172.21.15.94:0/145175936 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c067e30 msgr2=0x7fc36c0a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:39.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.912+0000 7fc371a57700 1 --2- 172.21.15.94:0/145175936 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c067e30 0x7fc36c0a1d70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0099e0 tx=0x7fc35c009250 comp rx=0 tx=0).stop 2023-09-30T00:22:39.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.912+0000 7fc371a57700 1 -- 172.21.15.94:0/145175936 shutdown_connections 2023-09-30T00:22:39.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.912+0000 7fc371a57700 1 --2- 172.21.15.94:0/145175936 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c067e30 0x7fc36c0a1d70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:39.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.912+0000 7fc371a57700 1 -- 172.21.15.94:0/145175936 >> 172.21.15.94:0/145175936 conn(0x7fc36c0702f0 msgr2=0x7fc36c0706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:39.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.912+0000 7fc371a57700 1 -- 172.21.15.94:0/145175936 shutdown_connections 2023-09-30T00:22:39.916 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.912+0000 7fc371a57700 1 -- 172.21.15.94:0/145175936 wait complete. 2023-09-30T00:22:39.919 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.913+0000 7fc371a57700 1 Processor -- start 2023-09-30T00:22:39.919 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.914+0000 7fc371a57700 1 -- start start 2023-09-30T00:22:39.919 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.914+0000 7fc371a57700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc36c067e30 0x7fc36c0a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:39.920 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.914+0000 7fc371a57700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc36c0a94d0 0x7fc36c0a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:39.920 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.914+0000 7fc371a57700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c0a5f20 0x7fc36c153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:39.920 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.914+0000 7fc371a57700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c0a5270 con 0x7fc36c067e30 2023-09-30T00:22:39.920 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.914+0000 7fc371a57700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c150740 con 0x7fc36c0a94d0 2023-09-30T00:22:39.920 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.914+0000 7fc371a57700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c150a00 con 0x7fc36c0a5f20 2023-09-30T00:22:39.920 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.914+0000 7fc36affd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc36c067e30 0x7fc36c0a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:39.920 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.914+0000 7fc36affd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc36c067e30 0x7fc36c0a4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52646/0 (socket says 172.21.15.94:52646) 2023-09-30T00:22:39.920 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.914+0000 7fc36b7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c0a5f20 0x7fc36c153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:39.920 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.914+0000 7fc36affd700 1 -- 172.21.15.94:0/2974339760 learned_addr learned my addr 172.21.15.94:0/2974339760 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:39.920 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.914+0000 7fc3627fc700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc36c0a94d0 0x7fc36c0a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:39.921 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.915+0000 7fc36affd700 1 -- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c0a5f20 msgr2=0x7fc36c153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:39.921 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.915+0000 7fc36affd700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c0a5f20 0x7fc36c153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:39.921 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.915+0000 7fc36affd700 1 -- 172.21.15.94:0/2974339760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc36c0a94d0 msgr2=0x7fc36c0a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:39.921 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.915+0000 7fc36affd700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc36c0a94d0 0x7fc36c0a59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:39.921 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.915+0000 7fc36affd700 1 -- 172.21.15.94:0/2974339760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c0094f0 con 0x7fc36c067e30 2023-09-30T00:22:39.921 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.915+0000 7fc36b7fe700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c0a5f20 0x7fc36c153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:39.921 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.915+0000 7fc36affd700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc36c067e30 0x7fc36c0a4830 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0094c0 tx=0x7fc35c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:39.921 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.915+0000 7fc3637fe700 1 -- 172.21.15.94:0/2974339760 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c01f070 con 0x7fc36c067e30 2023-09-30T00:22:39.921 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.915+0000 7fc3637fe700 1 -- 172.21.15.94:0/2974339760 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c011e80 con 0x7fc36c067e30 2023-09-30T00:22:39.921 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.915+0000 7fc371a57700 1 -- 172.21.15.94:0/2974339760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c144210 con 0x7fc36c067e30 2023-09-30T00:22:39.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.915+0000 7fc3637fe700 1 -- 172.21.15.94:0/2974339760 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c01a650 con 0x7fc36c067e30 2023-09-30T00:22:39.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.915+0000 7fc371a57700 1 -- 172.21.15.94:0/2974339760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36c143430 con 0x7fc36c067e30 2023-09-30T00:22:39.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.916+0000 7fc371a57700 1 -- 172.21.15.94:0/2974339760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36c06fbd0 con 0x7fc36c067e30 2023-09-30T00:22:39.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.917+0000 7fc3637fe700 1 -- 172.21.15.94:0/2974339760 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fc35c01a9c0 con 0x7fc36c067e30 2023-09-30T00:22:39.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.917+0000 7fc3637fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:39.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.917+0000 7fc3637fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:39.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.917+0000 7fc3637fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:39.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.917+0000 7fc3637fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:39.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.917+0000 7fc3637fe700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc34c0987b0 0x7fc34c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:39.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.917+0000 7fc3627fc700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc34c0987b0 0x7fc34c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:39.922 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.917+0000 7fc3637fe700 1 -- 172.21.15.94:0/2974339760 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fc35c0c1a70 con 0x7fc36c067e30 2023-09-30T00:22:39.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.918+0000 7fc3627fc700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc34c0987b0 0x7fc34c09ac60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc36c13c610 tx=0x7fc354007ec0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:39.923 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:39.921+0000 7fc3637fe700 1 -- 172.21.15.94:0/2974339760 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc35c08f140 con 0x7fc36c067e30 2023-09-30T00:22:40.150 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.149+0000 7fc371a57700 1 -- 172.21.15.94:0/2974339760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc36c0a8df0 con 0x7fc36c067e30 2023-09-30T00:22:40.151 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.149+0000 7fc3637fe700 1 -- 172.21.15.94:0/2974339760 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc35c092760 con 0x7fc36c067e30 2023-09-30T00:22:40.151 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:40.151 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:40.152 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 10 mgrc shutdown 2023-09-30T00:22:40.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 1 -- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc34c0987b0 msgr2=0x7fc34c09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:40.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc34c0987b0 0x7fc34c09ac60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc36c13c610 tx=0x7fc354007ec0 comp rx=0 tx=0).stop 2023-09-30T00:22:40.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 1 -- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc36c067e30 msgr2=0x7fc36c0a4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:40.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc36c067e30 0x7fc36c0a4830 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0094c0 tx=0x7fc35c017000 comp rx=0 tx=0).stop 2023-09-30T00:22:40.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 1 -- 172.21.15.94:0/2974339760 shutdown_connections 2023-09-30T00:22:40.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fc34c0987b0 0x7fc34c09ac60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:40.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc36c0a5f20 0x7fc36c153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:40.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc36c067e30 0x7fc36c0a4830 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:40.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 1 --2- 172.21.15.94:0/2974339760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc36c0a94d0 0x7fc36c0a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:40.153 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 1 -- 172.21.15.94:0/2974339760 >> 172.21.15.94:0/2974339760 conn(0x7fc36c0702f0 msgr2=0x7fc36c0b6680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:40.154 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 1 -- 172.21.15.94:0/2974339760 shutdown_connections 2023-09-30T00:22:40.154 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:40.151+0000 7fc371a57700 1 -- 172.21.15.94:0/2974339760 wait complete. 2023-09-30T00:22:43.165 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:43.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.285+0000 7f4744113700 1 Processor -- start 2023-09-30T00:22:43.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.285+0000 7f4744113700 1 -- start start 2023-09-30T00:22:43.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.285+0000 7f4744113700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f473c0b7160 0x7f473c0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:43.289 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.285+0000 7f4744113700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473c0a1310 con 0x7f473c13fd70 2023-09-30T00:22:43.290 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.285+0000 7f4744113700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473c071020 con 0x7f473c070c50 2023-09-30T00:22:43.290 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.285+0000 7f4744113700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473c0b76a0 con 0x7f473c0b7160 2023-09-30T00:22:43.290 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f47426b0700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f473c13fd70 0x7f473c14b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40038/0 (socket says 172.21.15.94:40038) 2023-09-30T00:22:43.290 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f47426b0700 1 -- 172.21.15.94:0/3104141242 learned_addr learned my addr 172.21.15.94:0/3104141242 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:22:43.290 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f47416ae700 1 --2- 172.21.15.94:0/3104141242 >> v2:172.21.15.121:3300/0 conn(0x7f473c0b7160 0x7f473c0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:43.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f4740ead700 1 -- 172.21.15.94:0/3104141242 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753396679 0 0) 0x7f4734001490 con 0x7f473c13fd70 2023-09-30T00:22:43.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f4740ead700 1 -- 172.21.15.94:0/3104141242 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4728003570 con 0x7f473c13fd70 2023-09-30T00:22:43.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f47416ae700 1 -- 172.21.15.94:0/3104141242 >> v1:172.21.15.121:6789/0 conn(0x7f473c070c50 legacy=0x7f473c0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:43.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f47416ae700 1 -- 172.21.15.94:0/3104141242 >> v1:172.21.15.94:6789/0 conn(0x7f473c13fd70 legacy=0x7f473c14b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:43.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f4740ead700 1 -- 172.21.15.94:0/3104141242 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3341196723 0 0) 0x7f4728003570 con 0x7f473c13fd70 2023-09-30T00:22:43.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f47416ae700 1 -- 172.21.15.94:0/3104141242 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c0b7940 con 0x7f473c0b7160 2023-09-30T00:22:43.291 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f47416ae700 1 --2- 172.21.15.94:0/3104141242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c0b7160 0x7f473c0a0dd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f472c0098e0 tx=0x7f472c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aa8367c88833b8c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:43.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f4740ead700 1 -- 172.21.15.94:0/3104141242 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734001980 con 0x7f473c0b7160 2023-09-30T00:22:43.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f4740ead700 1 -- 172.21.15.94:0/3104141242 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472c011420 con 0x7f473c0b7160 2023-09-30T00:22:43.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.286+0000 7f4740ead700 1 -- 172.21.15.94:0/3104141242 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472c0117a0 con 0x7f473c0b7160 2023-09-30T00:22:43.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.287+0000 7f4744113700 1 -- 172.21.15.94:0/3104141242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c0b7160 msgr2=0x7f473c0a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:43.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.287+0000 7f4744113700 1 --2- 172.21.15.94:0/3104141242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c0b7160 0x7f473c0a0dd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f472c0098e0 tx=0x7f472c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:22:43.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.287+0000 7f4744113700 1 -- 172.21.15.94:0/3104141242 shutdown_connections 2023-09-30T00:22:43.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.287+0000 7f4744113700 1 --2- 172.21.15.94:0/3104141242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c0b7160 0x7f473c0a0dd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:43.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.287+0000 7f4744113700 1 -- 172.21.15.94:0/3104141242 >> 172.21.15.94:0/3104141242 conn(0x7f473c06a090 msgr2=0x7f473c0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:43.292 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.287+0000 7f4744113700 1 -- 172.21.15.94:0/3104141242 shutdown_connections 2023-09-30T00:22:43.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.287+0000 7f4744113700 1 -- 172.21.15.94:0/3104141242 wait complete. 2023-09-30T00:22:43.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.288+0000 7f4744113700 1 Processor -- start 2023-09-30T00:22:43.293 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.288+0000 7f4744113700 1 -- start start 2023-09-30T00:22:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.288+0000 7f4744113700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f473c070c50 0x7f473c0b16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.288+0000 7f4744113700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c0b7160 0x7f473c153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.288+0000 7f4744113700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f473c13fd70 0x7f473c154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.288+0000 7f4744113700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c1513d0 con 0x7f473c13fd70 2023-09-30T00:22:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.288+0000 7f4744113700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c151670 con 0x7f473c0b7160 2023-09-30T00:22:43.296 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.288+0000 7f4744113700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c0adf20 con 0x7f473c070c50 2023-09-30T00:22:43.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.288+0000 7f4741eaf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f473c070c50 0x7f473c0b16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:43.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f47426b0700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f473c13fd70 0x7f473c154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:43.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4741eaf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f473c070c50 0x7f473c0b16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:36260/0 (socket says 172.21.15.94:36260) 2023-09-30T00:22:43.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f47426b0700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f473c13fd70 0x7f473c154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52650/0 (socket says 172.21.15.94:52650) 2023-09-30T00:22:43.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4741eaf700 1 -- 172.21.15.94:0/4222426969 learned_addr learned my addr 172.21.15.94:0/4222426969 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:43.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f47416ae700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c0b7160 0x7f473c153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:43.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4741eaf700 1 -- 172.21.15.94:0/4222426969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c0b7160 msgr2=0x7f473c153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:43.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4741eaf700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c0b7160 0x7f473c153cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:43.297 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4741eaf700 1 -- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f473c13fd70 msgr2=0x7f473c154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:43.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4741eaf700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f473c13fd70 0x7f473c154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:43.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4741eaf700 1 -- 172.21.15.94:0/4222426969 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c009420 con 0x7f473c070c50 2023-09-30T00:22:43.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4741eaf700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f473c070c50 0x7f473c0b16e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f473800c790 tx=0x7f4738007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:43.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4731ffb700 1 -- 172.21.15.94:0/4222426969 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738011590 con 0x7f473c070c50 2023-09-30T00:22:43.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4731ffb700 1 -- 172.21.15.94:0/4222426969 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4738011710 con 0x7f473c070c50 2023-09-30T00:22:43.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4744113700 1 -- 172.21.15.94:0/4222426969 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473c0b01a0 con 0x7f473c070c50 2023-09-30T00:22:43.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4731ffb700 1 -- 172.21.15.94:0/4222426969 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738011ad0 con 0x7f473c070c50 2023-09-30T00:22:43.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.289+0000 7f4744113700 1 -- 172.21.15.94:0/4222426969 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473c0b1de0 con 0x7f473c070c50 2023-09-30T00:22:43.298 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.290+0000 7f4744113700 1 -- 172.21.15.94:0/4222426969 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f473c071630 con 0x7f473c070c50 2023-09-30T00:22:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.291+0000 7f4731ffb700 1 -- 172.21.15.94:0/4222426969 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f473801b3b0 con 0x7f473c070c50 2023-09-30T00:22:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.291+0000 7f4731ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.291+0000 7f4731ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.291+0000 7f4731ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.291+0000 7f4731ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.291+0000 7f4731ffb700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f47200987b0 0x7f472009ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.292+0000 7f47416ae700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f47200987b0 0x7f472009ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.292+0000 7f4731ffb700 1 -- 172.21.15.94:0/4222426969 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f47380bfa30 con 0x7f473c070c50 2023-09-30T00:22:43.299 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.292+0000 7f47416ae700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f47200987b0 0x7f472009ac60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f472c009fd0 tx=0x7f472c016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.296+0000 7f4731ffb700 1 -- 172.21.15.94:0/4222426969 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f473808cfd0 con 0x7f473c070c50 2023-09-30T00:22:43.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.516+0000 7f4744113700 1 -- 172.21.15.94:0/4222426969 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f473c0ae060 con 0x7f473c070c50 2023-09-30T00:22:43.518 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.517+0000 7f4731ffb700 1 -- 172.21.15.94:0/4222426969 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f473c0ae060 con 0x7f473c070c50 2023-09-30T00:22:43.519 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:43.519 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:43.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 10 mgrc shutdown 2023-09-30T00:22:43.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 1 -- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f47200987b0 msgr2=0x7f472009ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:43.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f47200987b0 0x7f472009ac60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f472c009fd0 tx=0x7f472c016000 comp rx=0 tx=0).stop 2023-09-30T00:22:43.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 1 -- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f473c070c50 msgr2=0x7f473c0b16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:43.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f473c070c50 0x7f473c0b16e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f473800c790 tx=0x7f4738007f60 comp rx=0 tx=0).stop 2023-09-30T00:22:43.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 1 -- 172.21.15.94:0/4222426969 shutdown_connections 2023-09-30T00:22:43.521 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f47200987b0 0x7f472009ac60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:43.522 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f473c070c50 0x7f473c0b16e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:43.522 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f473c13fd70 0x7f473c154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:43.522 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 1 --2- 172.21.15.94:0/4222426969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c0b7160 0x7f473c153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:43.522 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 1 -- 172.21.15.94:0/4222426969 >> 172.21.15.94:0/4222426969 conn(0x7f473c06a090 msgr2=0x7f473c13d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:43.522 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 1 -- 172.21.15.94:0/4222426969 shutdown_connections 2023-09-30T00:22:43.522 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:43.519+0000 7f4744113700 1 -- 172.21.15.94:0/4222426969 wait complete. 2023-09-30T00:22:46.533 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:47.010 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.007+0000 7fcfe57a6700 1 Processor -- start 2023-09-30T00:22:47.010 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.007+0000 7fcfe57a6700 1 -- start start 2023-09-30T00:22:47.010 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfe57a6700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcfe00a1080 0x7fcfe00a2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:47.010 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfe57a6700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe00b7220 con 0x7fcfe00b6e50 2023-09-30T00:22:47.010 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfe57a6700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe00a14a0 con 0x7fcfe013fe20 2023-09-30T00:22:47.010 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfe57a6700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe00a36c0 con 0x7fcfe00a1080 2023-09-30T00:22:47.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfdffff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcfe00b6e50 0x7fcfe00a2fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46454/0 (socket says 172.21.15.94:46454) 2023-09-30T00:22:47.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfdffff700 1 -- 172.21.15.94:0/1648281608 learned_addr learned my addr 172.21.15.94:0/1648281608 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:22:47.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfdf7fe700 1 --2- 172.21.15.94:0/1648281608 >> v2:172.21.15.94:3300/0 conn(0x7fcfe00a1080 0x7fcfe00a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:47.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfdf7fe700 1 -- 172.21.15.94:0/1648281608 >> v1:172.21.15.121:6789/0 conn(0x7fcfe013fe20 legacy=0x7fcfe014b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:47.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfdf7fe700 1 -- 172.21.15.94:0/1648281608 >> v1:172.21.15.94:6789/0 conn(0x7fcfe00b6e50 legacy=0x7fcfe00a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:47.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfdf7fe700 1 -- 172.21.15.94:0/1648281608 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe00a2870 con 0x7fcfe00a1080 2023-09-30T00:22:47.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfdeffd700 1 -- 172.21.15.94:0/1648281608 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506838171 0 0) 0x7fcfe00b7220 con 0x7fcfe00b6e50 2023-09-30T00:22:47.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfdf7fe700 1 --2- 172.21.15.94:0/1648281608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfe00a1080 0x7fcfe00a2140 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40098e0 tx=0x7fcfd40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df35d1970fd291be server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:47.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfdeffd700 1 -- 172.21.15.94:0/1648281608 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4009e80 con 0x7fcfe00a1080 2023-09-30T00:22:47.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfdeffd700 1 -- 172.21.15.94:0/1648281608 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd400b820 con 0x7fcfe00a1080 2023-09-30T00:22:47.011 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.008+0000 7fcfdeffd700 1 -- 172.21.15.94:0/1648281608 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400bba0 con 0x7fcfe00a1080 2023-09-30T00:22:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.009+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1648281608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfe00a1080 msgr2=0x7fcfe00a2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.009+0000 7fcfe57a6700 1 --2- 172.21.15.94:0/1648281608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfe00a1080 0x7fcfe00a2140 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40098e0 tx=0x7fcfd40091a0 comp rx=0 tx=0).stop 2023-09-30T00:22:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.009+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1648281608 shutdown_connections 2023-09-30T00:22:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.009+0000 7fcfe57a6700 1 --2- 172.21.15.94:0/1648281608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfe00a1080 0x7fcfe00a2140 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.009+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1648281608 >> 172.21.15.94:0/1648281608 conn(0x7fcfe00702f0 msgr2=0x7fcfe00706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.009+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1648281608 shutdown_connections 2023-09-30T00:22:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.009+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1648281608 wait complete. 2023-09-30T00:22:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.010+0000 7fcfe57a6700 1 Processor -- start 2023-09-30T00:22:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.010+0000 7fcfe57a6700 1 -- start start 2023-09-30T00:22:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe57a6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcfe00a1080 0x7fcfe00b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:47.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe57a6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfe00b6e50 0x7fcfe0143610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:47.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe57a6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfe013fe20 0x7fcfe013a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:47.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe57a6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00a4f50 con 0x7fcfe013fe20 2023-09-30T00:22:47.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe57a6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00a3ea0 con 0x7fcfe00b6e50 2023-09-30T00:22:47.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe57a6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00a4160 con 0x7fcfe00a1080 2023-09-30T00:22:47.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfdffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcfe00a1080 0x7fcfe00b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:47.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe4fa5700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfe013fe20 0x7fcfe013a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:47.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfdffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcfe00a1080 0x7fcfe00b5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:36848/0 (socket says 172.21.15.94:36848) 2023-09-30T00:22:47.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfdffff700 1 -- 172.21.15.94:0/1986281148 learned_addr learned my addr 172.21.15.94:0/1986281148 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:47.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfdf7fe700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfe00b6e50 0x7fcfe0143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:47.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe4fa5700 1 -- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcfe00a1080 msgr2=0x7fcfe00b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:47.016 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe4fa5700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcfe00a1080 0x7fcfe00b5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe4fa5700 1 -- 172.21.15.94:0/1986281148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfe00b6e50 msgr2=0x7fcfe0143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe4fa5700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfe00b6e50 0x7fcfe0143610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe4fa5700 1 -- 172.21.15.94:0/1986281148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd4009420 con 0x7fcfe013fe20 2023-09-30T00:22:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfdffff700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcfe00a1080 0x7fcfe00b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe4fa5700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfe013fe20 0x7fcfe013a3a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800bad0 tx=0x7fcfd8009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfc7fff700 1 -- 172.21.15.94:0/1986281148 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd800c760 con 0x7fcfe013fe20 2023-09-30T00:22:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfc7fff700 1 -- 172.21.15.94:0/1986281148 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd800c8c0 con 0x7fcfe013fe20 2023-09-30T00:22:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.011+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1986281148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe0142a30 con 0x7fcfe013fe20 2023-09-30T00:22:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.012+0000 7fcfc7fff700 1 -- 172.21.15.94:0/1986281148 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd800cc80 con 0x7fcfe013fe20 2023-09-30T00:22:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.012+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1986281148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe01440c0 con 0x7fcfe013fe20 2023-09-30T00:22:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.012+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1986281148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe0066e50 con 0x7fcfe013fe20 2023-09-30T00:22:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.014+0000 7fcfc7fff700 1 -- 172.21.15.94:0/1986281148 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fcfd8014620 con 0x7fcfe013fe20 2023-09-30T00:22:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.014+0000 7fcfc7fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.014+0000 7fcfc7fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.014+0000 7fcfc7fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.014+0000 7fcfc7fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.014+0000 7fcfc7fff700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcfcc0987b0 0x7fcfcc09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.015+0000 7fcfdffff700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcfcc0987b0 0x7fcfcc09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.015+0000 7fcfc7fff700 1 -- 172.21.15.94:0/1986281148 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fcfd80c0a50 con 0x7fcfe013fe20 2023-09-30T00:22:47.019 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.015+0000 7fcfdffff700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcfcc0987b0 0x7fcfcc09ac60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0007fd0 tx=0x7fcfd0007df0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.018+0000 7fcfc7fff700 1 -- 172.21.15.94:0/1986281148 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcfd808e090 con 0x7fcfe013fe20 2023-09-30T00:22:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.239+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1986281148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcfe00871f0 con 0x7fcfe013fe20 2023-09-30T00:22:47.241 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.240+0000 7fcfc7fff700 1 -- 172.21.15.94:0/1986281148 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcfd8022070 con 0x7fcfe013fe20 2023-09-30T00:22:47.241 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:47.242 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.241+0000 7fcfe57a6700 10 mgrc shutdown 2023-09-30T00:22:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.241+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcfcc0987b0 msgr2=0x7fcfcc09ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.241+0000 7fcfe57a6700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcfcc0987b0 0x7fcfcc09ac60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0007fd0 tx=0x7fcfd0007df0 comp rx=0 tx=0).stop 2023-09-30T00:22:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.241+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfe013fe20 msgr2=0x7fcfe013a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.241+0000 7fcfe57a6700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfe013fe20 0x7fcfe013a3a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800bad0 tx=0x7fcfd8009760 comp rx=0 tx=0).stop 2023-09-30T00:22:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.242+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1986281148 shutdown_connections 2023-09-30T00:22:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.242+0000 7fcfe57a6700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcfcc0987b0 0x7fcfcc09ac60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.242+0000 7fcfe57a6700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcfe00a1080 0x7fcfe00b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.242+0000 7fcfe57a6700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfe013fe20 0x7fcfe013a3a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.242+0000 7fcfe57a6700 1 --2- 172.21.15.94:0/1986281148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfe00b6e50 0x7fcfe0143610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.242+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1986281148 >> 172.21.15.94:0/1986281148 conn(0x7fcfe00702f0 msgr2=0x7fcfe0067ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.242+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1986281148 shutdown_connections 2023-09-30T00:22:47.244 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:47.242+0000 7fcfe57a6700 1 -- 172.21.15.94:0/1986281148 wait complete. 2023-09-30T00:22:50.256 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:50.381 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.380+0000 7efef2c5d700 1 Processor -- start 2023-09-30T00:22:50.381 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.380+0000 7efef2c5d700 1 -- start start 2023-09-30T00:22:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef2c5d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efeec067e30 0x7efeec0a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef2c5d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efeec0a94d0 0x7efeec14b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef2c5d700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec068200 con 0x7efeec0a9ea0 2023-09-30T00:22:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef2c5d700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeec0a22b0 con 0x7efeec0a94d0 2023-09-30T00:22:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef2c5d700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeec0a24a0 con 0x7efeec067e30 2023-09-30T00:22:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef11fa700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efeec0a94d0 0x7efeec14b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef11fa700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efeec0a94d0 0x7efeec14b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60010/0 (socket says 172.21.15.94:60010) 2023-09-30T00:22:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef11fa700 1 -- 172.21.15.94:0/207183653 learned_addr learned my addr 172.21.15.94:0/207183653 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef09f9700 1 --2- 172.21.15.94:0/207183653 >> v2:172.21.15.121:3300/0 conn(0x7efeec067e30 0x7efeec0a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef11fa700 1 -- 172.21.15.94:0/207183653 >> v2:172.21.15.121:3300/0 conn(0x7efeec067e30 msgr2=0x7efeec0a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef11fa700 1 --2- 172.21.15.94:0/207183653 >> v2:172.21.15.121:3300/0 conn(0x7efeec067e30 0x7efeec0a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef11fa700 1 -- 172.21.15.94:0/207183653 >> v1:172.21.15.121:6789/0 conn(0x7efeec0a9ea0 legacy=0x7efeec13fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.381+0000 7efef11fa700 1 -- 172.21.15.94:0/207183653 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec0a2740 con 0x7efeec0a94d0 2023-09-30T00:22:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.382+0000 7efef11fa700 1 --2- 172.21.15.94:0/207183653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efeec0a94d0 0x7efeec14b060 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efeec0688e0 tx=0x7efee4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf1143471089cf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.382+0000 7efee37fe700 1 -- 172.21.15.94:0/207183653 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee400e070 con 0x7efeec0a94d0 2023-09-30T00:22:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.382+0000 7efee37fe700 1 -- 172.21.15.94:0/207183653 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee400ba10 con 0x7efeec0a94d0 2023-09-30T00:22:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.382+0000 7efee37fe700 1 -- 172.21.15.94:0/207183653 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee40092e0 con 0x7efeec0a94d0 2023-09-30T00:22:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.382+0000 7efef2c5d700 1 -- 172.21.15.94:0/207183653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efeec0a94d0 msgr2=0x7efeec14b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.382+0000 7efef2c5d700 1 --2- 172.21.15.94:0/207183653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efeec0a94d0 0x7efeec14b060 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efeec0688e0 tx=0x7efee4009840 comp rx=0 tx=0).stop 2023-09-30T00:22:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.382+0000 7efef2c5d700 1 -- 172.21.15.94:0/207183653 shutdown_connections 2023-09-30T00:22:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.382+0000 7efef2c5d700 1 --2- 172.21.15.94:0/207183653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efeec0a94d0 0x7efeec14b060 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.382+0000 7efef2c5d700 1 --2- 172.21.15.94:0/207183653 >> v2:172.21.15.121:3300/0 conn(0x7efeec067e30 0x7efeec0a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.382+0000 7efef2c5d700 1 -- 172.21.15.94:0/207183653 >> 172.21.15.94:0/207183653 conn(0x7efeec0702f0 msgr2=0x7efeec0706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.382+0000 7efef2c5d700 1 -- 172.21.15.94:0/207183653 shutdown_connections 2023-09-30T00:22:50.386 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.382+0000 7efef2c5d700 1 -- 172.21.15.94:0/207183653 wait complete. 2023-09-30T00:22:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.383+0000 7efef2c5d700 1 Processor -- start 2023-09-30T00:22:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef2c5d700 1 -- start start 2023-09-30T00:22:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef2c5d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeec067e30 0x7efeec143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef2c5d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efeec0a94d0 0x7efeec1428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:50.388 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef2c5d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efeec096750 0x7efeec142df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef2c5d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec143330 con 0x7efeec0a94d0 2023-09-30T00:22:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef2c5d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec144700 con 0x7efeec067e30 2023-09-30T00:22:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef2c5d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec143540 con 0x7efeec096750 2023-09-30T00:22:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef11fa700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efeec096750 0x7efeec142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efee3fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efeec0a94d0 0x7efeec1428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef11fa700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efeec096750 0x7efeec142df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:36856/0 (socket says 172.21.15.94:36856) 2023-09-30T00:22:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef11fa700 1 -- 172.21.15.94:0/2792017478 learned_addr learned my addr 172.21.15.94:0/2792017478 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef09f9700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeec067e30 0x7efeec143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef11fa700 1 -- 172.21.15.94:0/2792017478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeec067e30 msgr2=0x7efeec143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef11fa700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeec067e30 0x7efeec143d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef11fa700 1 -- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efeec0a94d0 msgr2=0x7efeec1428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.384+0000 7efef11fa700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efeec0a94d0 0x7efeec1428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.385+0000 7efef11fa700 1 -- 172.21.15.94:0/2792017478 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee4009ae0 con 0x7efeec096750 2023-09-30T00:22:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.385+0000 7efee3fff700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efeec0a94d0 0x7efeec1428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.385+0000 7efef11fa700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efeec096750 0x7efeec142df0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efee4009880 tx=0x7efee4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.385+0000 7efee0ff9700 1 -- 172.21.15.94:0/2792017478 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee40092e0 con 0x7efeec096750 2023-09-30T00:22:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.385+0000 7efee0ff9700 1 -- 172.21.15.94:0/2792017478 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee400be80 con 0x7efeec096750 2023-09-30T00:22:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.385+0000 7efef2c5d700 1 -- 172.21.15.94:0/2792017478 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeec143800 con 0x7efeec096750 2023-09-30T00:22:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.385+0000 7efee0ff9700 1 -- 172.21.15.94:0/2792017478 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee401d420 con 0x7efeec096750 2023-09-30T00:22:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.385+0000 7efef2c5d700 1 -- 172.21.15.94:0/2792017478 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeec0a6910 con 0x7efeec096750 2023-09-30T00:22:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.386+0000 7efef2c5d700 1 -- 172.21.15.94:0/2792017478 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeec06fbd0 con 0x7efeec096750 2023-09-30T00:22:50.390 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.387+0000 7efee0ff9700 1 -- 172.21.15.94:0/2792017478 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7efee401d810 con 0x7efeec096750 2023-09-30T00:22:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.387+0000 7efee0ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.387+0000 7efee0ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.387+0000 7efee0ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.387+0000 7efee0ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.387+0000 7efee0ff9700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7efed8098780 0x7efed809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.387+0000 7efee0ff9700 1 -- 172.21.15.94:0/2792017478 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7efee40c1b50 con 0x7efeec096750 2023-09-30T00:22:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.387+0000 7efef09f9700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7efed8098780 0x7efed809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:50.391 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.388+0000 7efef09f9700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7efed8098780 0x7efed809ac30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efee8006fd0 tx=0x7efee8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:50.392 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.391+0000 7efee0ff9700 1 -- 172.21.15.94:0/2792017478 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efee408f220 con 0x7efeec096750 2023-09-30T00:22:50.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.614+0000 7efef2c5d700 1 -- 172.21.15.94:0/2792017478 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efeec1425a0 con 0x7efeec096750 2023-09-30T00:22:50.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.614+0000 7efee0ff9700 1 -- 172.21.15.94:0/2792017478 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7efee4092840 con 0x7efeec096750 2023-09-30T00:22:50.615 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:50.615 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:50.617 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.616+0000 7efef2c5d700 10 mgrc shutdown 2023-09-30T00:22:50.617 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.616+0000 7efef2c5d700 1 -- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7efed8098780 msgr2=0x7efed809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:50.617 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.616+0000 7efef2c5d700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7efed8098780 0x7efed809ac30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efee8006fd0 tx=0x7efee8008040 comp rx=0 tx=0).stop 2023-09-30T00:22:50.617 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.616+0000 7efef2c5d700 1 -- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efeec096750 msgr2=0x7efeec142df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:50.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.616+0000 7efef2c5d700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efeec096750 0x7efeec142df0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efee4009880 tx=0x7efee4017000 comp rx=0 tx=0).stop 2023-09-30T00:22:50.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.617+0000 7efef2c5d700 1 -- 172.21.15.94:0/2792017478 shutdown_connections 2023-09-30T00:22:50.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.617+0000 7efef2c5d700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7efed8098780 0x7efed809ac30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:50.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.617+0000 7efef2c5d700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efeec096750 0x7efeec142df0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:50.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.617+0000 7efef2c5d700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeec067e30 0x7efeec143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:50.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.617+0000 7efef2c5d700 1 --2- 172.21.15.94:0/2792017478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efeec0a94d0 0x7efeec1428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:50.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.617+0000 7efef2c5d700 1 -- 172.21.15.94:0/2792017478 >> 172.21.15.94:0/2792017478 conn(0x7efeec0702f0 msgr2=0x7efeec146840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:50.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.617+0000 7efef2c5d700 1 -- 172.21.15.94:0/2792017478 shutdown_connections 2023-09-30T00:22:50.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:50.617+0000 7efef2c5d700 1 -- 172.21.15.94:0/2792017478 wait complete. 2023-09-30T00:22:53.629 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.754+0000 7f302d541700 1 Processor -- start 2023-09-30T00:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.754+0000 7f302d541700 1 -- start start 2023-09-30T00:22:53.756 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.755+0000 7f302d541700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3028070c50 0x7f30280a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:53.756 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.755+0000 7f302d541700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30280a1310 con 0x7f30280b7160 2023-09-30T00:22:53.756 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.755+0000 7f302d541700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3028071020 con 0x7f302813fd70 2023-09-30T00:22:53.756 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.755+0000 7f302d541700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30280b76a0 con 0x7f3028070c50 2023-09-30T00:22:53.756 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.755+0000 7f3026ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3028070c50 0x7f30280a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.755+0000 7f30267fc700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f30280b7160 0x7f30280a2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46456/0 (socket says 172.21.15.94:46456) 2023-09-30T00:22:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.755+0000 7f30267fc700 1 -- 172.21.15.94:0/568500426 learned_addr learned my addr 172.21.15.94:0/568500426 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-09-30T00:22:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.755+0000 7f3026ffd700 1 -- 172.21.15.94:0/568500426 >> v1:172.21.15.121:6789/0 conn(0x7f302813fd70 legacy=0x7f302814b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.755+0000 7f3025ffb700 1 -- 172.21.15.94:0/568500426 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151917659 0 0) 0x7f3010001110 con 0x7f30280b7160 2023-09-30T00:22:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.755+0000 7f3026ffd700 1 -- 172.21.15.94:0/568500426 >> v1:172.21.15.94:6789/0 conn(0x7f30280b7160 legacy=0x7f30280a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.755+0000 7f3026ffd700 1 -- 172.21.15.94:0/568500426 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30280b7940 con 0x7f3028070c50 2023-09-30T00:22:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.756+0000 7f3026ffd700 1 --2- 172.21.15.94:0/568500426 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3028070c50 0x7f30280a0dd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3020009e90 tx=0x7f3020009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8ba93f8ffaa8f5d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.756+0000 7f3025ffb700 1 -- 172.21.15.94:0/568500426 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302000e070 con 0x7f3028070c50 2023-09-30T00:22:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.756+0000 7f3025ffb700 1 -- 172.21.15.94:0/568500426 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302000bc50 con 0x7f3028070c50 2023-09-30T00:22:53.757 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.756+0000 7f3025ffb700 1 -- 172.21.15.94:0/568500426 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3020012430 con 0x7f3028070c50 2023-09-30T00:22:53.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.756+0000 7f302d541700 1 -- 172.21.15.94:0/568500426 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3028070c50 msgr2=0x7f30280a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:53.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.756+0000 7f302d541700 1 --2- 172.21.15.94:0/568500426 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3028070c50 0x7f30280a0dd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3020009e90 tx=0x7f3020009780 comp rx=0 tx=0).stop 2023-09-30T00:22:53.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.756+0000 7f302d541700 1 -- 172.21.15.94:0/568500426 shutdown_connections 2023-09-30T00:22:53.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.756+0000 7f302d541700 1 --2- 172.21.15.94:0/568500426 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3028070c50 0x7f30280a0dd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:53.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.756+0000 7f302d541700 1 -- 172.21.15.94:0/568500426 >> 172.21.15.94:0/568500426 conn(0x7f302806a090 msgr2=0x7f30280702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:53.760 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.756+0000 7f302d541700 1 -- 172.21.15.94:0/568500426 shutdown_connections 2023-09-30T00:22:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.756+0000 7f302d541700 1 -- 172.21.15.94:0/568500426 wait complete. 2023-09-30T00:22:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.757+0000 7f302d541700 1 Processor -- start 2023-09-30T00:22:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.757+0000 7f302d541700 1 -- start start 2023-09-30T00:22:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f302d541700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3028070c50 0x7f302813b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f302d541700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30280b7160 0x7f302813aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f302d541700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f302813fd70 0x7f30281504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f302d541700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302813b140 con 0x7f3028070c50 2023-09-30T00:22:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f302d541700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3028151040 con 0x7f302813fd70 2023-09-30T00:22:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f302d541700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3028151300 con 0x7f30280b7160 2023-09-30T00:22:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f3026ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3028070c50 0x7f302813b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f30267fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30280b7160 0x7f302813aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:53.761 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f3026ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3028070c50 0x7f302813b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60022/0 (socket says 172.21.15.94:60022) 2023-09-30T00:22:53.762 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f3026ffd700 1 -- 172.21.15.94:0/2098669495 learned_addr learned my addr 172.21.15.94:0/2098669495 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:53.762 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f30277fe700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f302813fd70 0x7f30281504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:53.762 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f30267fc700 1 -- 172.21.15.94:0/2098669495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f302813fd70 msgr2=0x7f30281504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:53.762 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f30267fc700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f302813fd70 0x7f30281504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:53.762 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f30267fc700 1 -- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3028070c50 msgr2=0x7f302813b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:53.762 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f30267fc700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3028070c50 0x7f302813b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:53.762 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f30267fc700 1 -- 172.21.15.94:0/2098669495 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3020009a20 con 0x7f30280b7160 2023-09-30T00:22:53.762 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f3026ffd700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3028070c50 0x7f302813b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:53.762 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.758+0000 7f30267fc700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30280b7160 0x7f302813aae0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f301000c810 tx=0x7f3010007b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:53.762 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.759+0000 7f301effd700 1 -- 172.21.15.94:0/2098669495 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010005b30 con 0x7f30280b7160 2023-09-30T00:22:53.762 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.759+0000 7f301effd700 1 -- 172.21.15.94:0/2098669495 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3010005c90 con 0x7f30280b7160 2023-09-30T00:22:53.763 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.759+0000 7f302d541700 1 -- 172.21.15.94:0/2098669495 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3028150a00 con 0x7f30280b7160 2023-09-30T00:22:53.763 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.759+0000 7f301effd700 1 -- 172.21.15.94:0/2098669495 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30100072e0 con 0x7f30280b7160 2023-09-30T00:22:53.763 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.759+0000 7f302d541700 1 -- 172.21.15.94:0/2098669495 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3028143eb0 con 0x7f30280b7160 2023-09-30T00:22:53.763 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.760+0000 7f302d541700 1 -- 172.21.15.94:0/2098669495 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302813b2b0 con 0x7f30280b7160 2023-09-30T00:22:53.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.760+0000 7f301effd700 1 -- 172.21.15.94:0/2098669495 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f3010007690 con 0x7f30280b7160 2023-09-30T00:22:53.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.760+0000 7f301effd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:53.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.761+0000 7f301effd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:53.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.761+0000 7f301effd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:53.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.761+0000 7f301effd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:53.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.761+0000 7f301effd700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f30080987b0 0x7f300809ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:53.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.761+0000 7f3026ffd700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f30080987b0 0x7f300809ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:53.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.761+0000 7f301effd700 1 -- 172.21.15.94:0/2098669495 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f30100bf6e0 con 0x7f30280b7160 2023-09-30T00:22:53.764 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.761+0000 7f3026ffd700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f30080987b0 0x7f300809ac60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f30200125a0 tx=0x7f3020018000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:53.765 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.765+0000 7f301effd700 1 -- 172.21.15.94:0/2098669495 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f30100907a0 con 0x7f30280b7160 2023-09-30T00:22:53.986 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.986+0000 7f302d541700 1 -- 172.21.15.94:0/2098669495 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f30280a1310 con 0x7f30280b7160 2023-09-30T00:22:53.987 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.986+0000 7f301effd700 1 -- 172.21.15.94:0/2098669495 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f30280a1310 con 0x7f30280b7160 2023-09-30T00:22:53.987 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:53.987 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:53.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 10 mgrc shutdown 2023-09-30T00:22:53.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 1 -- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f30080987b0 msgr2=0x7f300809ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:53.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f30080987b0 0x7f300809ac60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f30200125a0 tx=0x7f3020018000 comp rx=0 tx=0).stop 2023-09-30T00:22:53.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 1 -- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30280b7160 msgr2=0x7f302813aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:53.989 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30280b7160 0x7f302813aae0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f301000c810 tx=0x7f3010007b90 comp rx=0 tx=0).stop 2023-09-30T00:22:53.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 1 -- 172.21.15.94:0/2098669495 shutdown_connections 2023-09-30T00:22:53.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f30080987b0 0x7f300809ac60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:53.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f302813fd70 0x7f30281504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:53.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3028070c50 0x7f302813b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:53.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 1 --2- 172.21.15.94:0/2098669495 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30280b7160 0x7f302813aae0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:53.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 1 -- 172.21.15.94:0/2098669495 >> 172.21.15.94:0/2098669495 conn(0x7f302806a090 msgr2=0x7f30280a8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:53.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 1 -- 172.21.15.94:0/2098669495 shutdown_connections 2023-09-30T00:22:53.990 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:53.988+0000 7f302d541700 1 -- 172.21.15.94:0/2098669495 wait complete. 2023-09-30T00:22:57.001 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:22:57.126 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.126+0000 7fe20ca79700 1 Processor -- start 2023-09-30T00:22:57.126 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.126+0000 7fe20ca79700 1 -- start start 2023-09-30T00:22:57.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe20ca79700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe2080a1080 0x7fe2080a2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:57.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe20ca79700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe20813fe20 0x7fe208140260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:57.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe20ca79700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2080b7220 con 0x7fe2080b6e50 2023-09-30T00:22:57.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe20ca79700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2081407a0 con 0x7fe20813fe20 2023-09-30T00:22:57.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe20ca79700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2080a36c0 con 0x7fe2080a1080 2023-09-30T00:22:57.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe207fff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe20813fe20 0x7fe208140260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:57.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe207fff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe20813fe20 0x7fe208140260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56556/0 (socket says 172.21.15.94:56556) 2023-09-30T00:22:57.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe207fff700 1 -- 172.21.15.94:0/4285237357 learned_addr learned my addr 172.21.15.94:0/4285237357 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:57.129 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe206ffd700 1 --2- 172.21.15.94:0/4285237357 >> v2:172.21.15.121:3300/0 conn(0x7fe2080a1080 0x7fe2080a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe207fff700 1 -- 172.21.15.94:0/4285237357 >> v2:172.21.15.121:3300/0 conn(0x7fe2080a1080 msgr2=0x7fe2080a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe207fff700 1 --2- 172.21.15.94:0/4285237357 >> v2:172.21.15.121:3300/0 conn(0x7fe2080a1080 0x7fe2080a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe207fff700 1 -- 172.21.15.94:0/4285237357 >> v1:172.21.15.121:6789/0 conn(0x7fe2080b6e50 legacy=0x7fe2080a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:22:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe207fff700 1 -- 172.21.15.94:0/4285237357 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2080a14a0 con 0x7fe20813fe20 2023-09-30T00:22:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe207fff700 1 --2- 172.21.15.94:0/4285237357 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe20813fe20 0x7fe208140260 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe2080712f0 tx=0x7fe200009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25df95fedc634136 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe2067fc700 1 -- 172.21.15.94:0/4285237357 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20000b820 con 0x7fe20813fe20 2023-09-30T00:22:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.127+0000 7fe2067fc700 1 -- 172.21.15.94:0/4285237357 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20000b980 con 0x7fe20813fe20 2023-09-30T00:22:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.128+0000 7fe2067fc700 1 -- 172.21.15.94:0/4285237357 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20000bd20 con 0x7fe20813fe20 2023-09-30T00:22:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.128+0000 7fe20ca79700 1 -- 172.21.15.94:0/4285237357 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe20813fe20 msgr2=0x7fe208140260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.128+0000 7fe20ca79700 1 --2- 172.21.15.94:0/4285237357 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe20813fe20 0x7fe208140260 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe2080712f0 tx=0x7fe200009250 comp rx=0 tx=0).stop 2023-09-30T00:22:57.130 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.128+0000 7fe20ca79700 1 -- 172.21.15.94:0/4285237357 shutdown_connections 2023-09-30T00:22:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.128+0000 7fe20ca79700 1 --2- 172.21.15.94:0/4285237357 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe20813fe20 0x7fe208140260 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.128+0000 7fe20ca79700 1 --2- 172.21.15.94:0/4285237357 >> v2:172.21.15.121:3300/0 conn(0x7fe2080a1080 0x7fe2080a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.128+0000 7fe20ca79700 1 -- 172.21.15.94:0/4285237357 >> 172.21.15.94:0/4285237357 conn(0x7fe2080702f0 msgr2=0x7fe2080706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.128+0000 7fe20ca79700 1 -- 172.21.15.94:0/4285237357 shutdown_connections 2023-09-30T00:22:57.131 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.128+0000 7fe20ca79700 1 -- 172.21.15.94:0/4285237357 wait complete. 2023-09-30T00:22:57.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.129+0000 7fe20ca79700 1 Processor -- start 2023-09-30T00:22:57.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.129+0000 7fe20ca79700 1 -- start start 2023-09-30T00:22:57.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe20ca79700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2080a1080 0x7fe208142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:57.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe20ca79700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2080b6e50 0x7fe2080a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:57.133 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe20ca79700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe20813fe20 0x7fe2080a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe20ca79700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2080a4d50 con 0x7fe20813fe20 2023-09-30T00:22:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe20ca79700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20813f5f0 con 0x7fe2080a1080 2023-09-30T00:22:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe20ca79700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2080b4ae0 con 0x7fe2080b6e50 2023-09-30T00:22:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe207fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe20813fe20 0x7fe2080a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe206ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2080b6e50 0x7fe2080a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe207fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe20813fe20 0x7fe2080a4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56566/0 (socket says 172.21.15.94:56566) 2023-09-30T00:22:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe207fff700 1 -- 172.21.15.94:0/1039592105 learned_addr learned my addr 172.21.15.94:0/1039592105 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:22:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe2077fe700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2080a1080 0x7fe208142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe206ffd700 1 -- 172.21.15.94:0/1039592105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2080a1080 msgr2=0x7fe208142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe206ffd700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2080a1080 0x7fe208142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:57.134 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe206ffd700 1 -- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe20813fe20 msgr2=0x7fe2080a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe206ffd700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe20813fe20 0x7fe2080a4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe206ffd700 1 -- 172.21.15.94:0/1039592105 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2000094f0 con 0x7fe2080b6e50 2023-09-30T00:22:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe207fff700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe20813fe20 0x7fe2080a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:22:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe206ffd700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2080b6e50 0x7fe2080a6220 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00d890 tx=0x7fe1fc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe1ef7fe700 1 -- 172.21.15.94:0/1039592105 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc0125f0 con 0x7fe2080b6e50 2023-09-30T00:22:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe1ef7fe700 1 -- 172.21.15.94:0/1039592105 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc012750 con 0x7fe2080b6e50 2023-09-30T00:22:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe1ef7fe700 1 -- 172.21.15.94:0/1039592105 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc012af0 con 0x7fe2080b6e50 2023-09-30T00:22:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.130+0000 7fe20ca79700 1 -- 172.21.15.94:0/1039592105 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe208143360 con 0x7fe2080b6e50 2023-09-30T00:22:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.131+0000 7fe20ca79700 1 -- 172.21.15.94:0/1039592105 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2080a5480 con 0x7fe2080b6e50 2023-09-30T00:22:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.132+0000 7fe20ca79700 1 -- 172.21.15.94:0/1039592105 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2080a4e90 con 0x7fe2080b6e50 2023-09-30T00:22:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.132+0000 7fe1ef7fe700 1 -- 172.21.15.94:0/1039592105 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fe1fc013480 con 0x7fe2080b6e50 2023-09-30T00:22:57.135 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.132+0000 7fe1ef7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:22:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.133+0000 7fe1ef7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:22:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.133+0000 7fe1ef7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:57.136 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.133+0000 7fe1ef7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:22:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.133+0000 7fe1ef7fe700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe1f40987b0 0x7fe1f409ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:22:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.133+0000 7fe2077fe700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe1f40987b0 0x7fe1f409ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:22:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.133+0000 7fe1ef7fe700 1 -- 172.21.15.94:0/1039592105 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fe1fc0c07a0 con 0x7fe2080b6e50 2023-09-30T00:22:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.134+0000 7fe2077fe700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe1f40987b0 0x7fe1f409ac60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8007fd0 tx=0x7fe1f80026a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:22:57.137 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.137+0000 7fe1ef7fe700 1 -- 172.21.15.94:0/1039592105 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe2080a4e90 con 0x7fe2080b6e50 2023-09-30T00:22:57.362 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.362+0000 7fe20ca79700 1 -- 172.21.15.94:0/1039592105 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe2080a4e90 con 0x7fe2080b6e50 2023-09-30T00:22:57.363 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.362+0000 7fe1ef7fe700 1 -- 172.21.15.94:0/1039592105 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe2080a4e90 con 0x7fe2080b6e50 2023-09-30T00:22:57.363 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:22:57.363 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:22:57.365 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 10 mgrc shutdown 2023-09-30T00:22:57.365 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 1 -- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe1f40987b0 msgr2=0x7fe1f409ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:57.365 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe1f40987b0 0x7fe1f409ac60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8007fd0 tx=0x7fe1f80026a0 comp rx=0 tx=0).stop 2023-09-30T00:22:57.365 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 1 -- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2080b6e50 msgr2=0x7fe2080a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:22:57.365 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2080b6e50 0x7fe2080a6220 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00d890 tx=0x7fe1fc008040 comp rx=0 tx=0).stop 2023-09-30T00:22:57.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 1 -- 172.21.15.94:0/1039592105 shutdown_connections 2023-09-30T00:22:57.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fe1f40987b0 0x7fe1f409ac60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:57.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe20813fe20 0x7fe2080a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:57.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2080a1080 0x7fe208142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:57.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 1 --2- 172.21.15.94:0/1039592105 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2080b6e50 0x7fe2080a6220 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:22:57.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 1 -- 172.21.15.94:0/1039592105 >> 172.21.15.94:0/1039592105 conn(0x7fe2080702f0 msgr2=0x7fe208067ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:22:57.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 1 -- 172.21.15.94:0/1039592105 shutdown_connections 2023-09-30T00:22:57.366 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:22:57.364+0000 7fe20ca79700 1 -- 172.21.15.94:0/1039592105 wait complete. 2023-09-30T00:23:00.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:23:00.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.502+0000 7f0a0d5fa700 1 Processor -- start 2023-09-30T00:23:00.502 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.502+0000 7f0a0d5fa700 1 -- start start 2023-09-30T00:23:00.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.502+0000 7f0a0d5fa700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a08070c50 0x7f0a080a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:00.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.502+0000 7f0a0d5fa700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0a080b7160 0x7f0a080a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:00.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.502+0000 7f0a0d5fa700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a080a1310 con 0x7f0a0813fd70 2023-09-30T00:23:00.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.502+0000 7f0a0d5fa700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a08071020 con 0x7f0a080b7160 2023-09-30T00:23:00.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.502+0000 7f0a0d5fa700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a080b76a0 con 0x7f0a08070c50 2023-09-30T00:23:00.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a06ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a08070c50 0x7f0a080a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:00.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a067fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0a080b7160 0x7f0a080a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:00.505 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a067fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0a080b7160 0x7f0a080a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56578/0 (socket says 172.21.15.94:56578) 2023-09-30T00:23:00.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a06ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a08070c50 0x7f0a080a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:43920/0 (socket says 172.21.15.94:43920) 2023-09-30T00:23:00.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a067fc700 1 -- 172.21.15.94:0/178432000 learned_addr learned my addr 172.21.15.94:0/178432000 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:23:00.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a06ffd700 1 -- 172.21.15.94:0/178432000 >> v2:172.21.15.94:3300/0 conn(0x7f0a080b7160 msgr2=0x7f0a080a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:23:00.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a06ffd700 1 --2- 172.21.15.94:0/178432000 >> v2:172.21.15.94:3300/0 conn(0x7f0a080b7160 0x7f0a080a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:00.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a06ffd700 1 -- 172.21.15.94:0/178432000 >> v1:172.21.15.121:6789/0 conn(0x7f0a0813fd70 legacy=0x7f0a080b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:23:00.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a06ffd700 1 -- 172.21.15.94:0/178432000 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a080b7940 con 0x7f0a08070c50 2023-09-30T00:23:00.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a067fc700 1 --2- 172.21.15.94:0/178432000 >> v2:172.21.15.94:3300/0 conn(0x7f0a080b7160 0x7f0a080a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:23:00.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a06ffd700 1 --2- 172.21.15.94:0/178432000 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a08070c50 0x7f0a080a0dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f09f80098e0 tx=0x7f09f80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0297784e4290175 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:00.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a05ffb700 1 -- 172.21.15.94:0/178432000 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f800b690 con 0x7f0a08070c50 2023-09-30T00:23:00.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a05ffb700 1 -- 172.21.15.94:0/178432000 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f800b7f0 con 0x7f0a08070c50 2023-09-30T00:23:00.506 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.503+0000 7f0a05ffb700 1 -- 172.21.15.94:0/178432000 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f800bb90 con 0x7f0a08070c50 2023-09-30T00:23:00.507 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.504+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/178432000 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a08070c50 msgr2=0x7f0a080a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:00.507 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.504+0000 7f0a0d5fa700 1 --2- 172.21.15.94:0/178432000 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a08070c50 0x7f0a080a0dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f09f80098e0 tx=0x7f09f80091a0 comp rx=0 tx=0).stop 2023-09-30T00:23:00.507 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.504+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/178432000 shutdown_connections 2023-09-30T00:23:00.507 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.504+0000 7f0a0d5fa700 1 --2- 172.21.15.94:0/178432000 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a08070c50 0x7f0a080a0dd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:00.507 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.504+0000 7f0a0d5fa700 1 --2- 172.21.15.94:0/178432000 >> v2:172.21.15.94:3300/0 conn(0x7f0a080b7160 0x7f0a080a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:00.507 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.504+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/178432000 >> 172.21.15.94:0/178432000 conn(0x7f0a0806a090 msgr2=0x7f0a080702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:23:00.507 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.504+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/178432000 shutdown_connections 2023-09-30T00:23:00.507 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.504+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/178432000 wait complete. 2023-09-30T00:23:00.509 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.505+0000 7f0a0d5fa700 1 Processor -- start 2023-09-30T00:23:00.510 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.505+0000 7f0a0d5fa700 1 -- start start 2023-09-30T00:23:00.510 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.505+0000 7f0a0d5fa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a08070c50 0x7f0a081409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:00.510 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.505+0000 7f0a0d5fa700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a080b7160 0x7f0a081540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:00.510 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.505+0000 7f0a0d5fa700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a0813fd70 0x7f0a08154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:00.510 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.505+0000 7f0a0d5fa700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a080ad2e0 con 0x7f0a080b7160 2023-09-30T00:23:00.510 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.505+0000 7f0a0d5fa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a080ad5c0 con 0x7f0a08070c50 2023-09-30T00:23:00.510 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.505+0000 7f0a0d5fa700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a08154c30 con 0x7f0a0813fd70 2023-09-30T00:23:00.510 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a067fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a080b7160 0x7f0a081540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:00.510 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a077fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a0813fd70 0x7f0a08154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:00.510 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a067fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a080b7160 0x7f0a081540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56580/0 (socket says 172.21.15.94:56580) 2023-09-30T00:23:00.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a067fc700 1 -- 172.21.15.94:0/2301839534 learned_addr learned my addr 172.21.15.94:0/2301839534 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:23:00.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a06ffd700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a08070c50 0x7f0a081409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:00.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a077fe700 1 -- 172.21.15.94:0/2301839534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a08070c50 msgr2=0x7f0a081409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:00.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a077fe700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a08070c50 0x7f0a081409e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:00.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a077fe700 1 -- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a080b7160 msgr2=0x7f0a081540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:00.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a077fe700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a080b7160 0x7f0a081540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:00.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a077fe700 1 -- 172.21.15.94:0/2301839534 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc0094f0 con 0x7f0a0813fd70 2023-09-30T00:23:00.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a077fe700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a0813fd70 0x7f0a08154610 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0a0000bad0 tx=0x7f0a00009760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:00.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f09eeffd700 1 -- 172.21.15.94:0/2301839534 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0000c760 con 0x7f0a0813fd70 2023-09-30T00:23:00.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f09eeffd700 1 -- 172.21.15.94:0/2301839534 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0000c8c0 con 0x7f0a0813fd70 2023-09-30T00:23:00.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/2301839534 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f8009420 con 0x7f0a0813fd70 2023-09-30T00:23:00.511 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f09eeffd700 1 -- 172.21.15.94:0/2301839534 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0000cc80 con 0x7f0a0813fd70 2023-09-30T00:23:00.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.506+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/2301839534 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a080adfb0 con 0x7f0a0813fd70 2023-09-30T00:23:00.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.507+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/2301839534 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a080a8470 con 0x7f0a0813fd70 2023-09-30T00:23:00.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.508+0000 7f09eeffd700 1 -- 172.21.15.94:0/2301839534 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f0a00014620 con 0x7f0a0813fd70 2023-09-30T00:23:00.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.508+0000 7f09eeffd700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:23:00.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.508+0000 7f09eeffd700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:23:00.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.508+0000 7f09eeffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:23:00.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.509+0000 7f09eeffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:23:00.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.509+0000 7f09eeffd700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f09f4098780 0x7f09f409ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:00.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.509+0000 7f0a06ffd700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f09f4098780 0x7f09f409ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:00.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.509+0000 7f09eeffd700 1 -- 172.21.15.94:0/2301839534 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f0a000c0ae0 con 0x7f0a0813fd70 2023-09-30T00:23:00.512 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.509+0000 7f0a06ffd700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f09f4098780 0x7f09f409ac30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f09f80030c0 tx=0x7f09f8016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:00.513 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.512+0000 7f09eeffd700 1 -- 172.21.15.94:0/2301839534 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0a0008e110 con 0x7f0a0813fd70 2023-09-30T00:23:00.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.732+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/2301839534 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0a080b7ad0 con 0x7f0a0813fd70 2023-09-30T00:23:00.733 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.733+0000 7f09eeffd700 1 -- 172.21.15.94:0/2301839534 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0a00022080 con 0x7f0a0813fd70 2023-09-30T00:23:00.733 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:23:00.734 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:23:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 10 mgrc shutdown 2023-09-30T00:23:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f09f4098780 msgr2=0x7f09f409ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f09f4098780 0x7f09f409ac30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f09f80030c0 tx=0x7f09f8016000 comp rx=0 tx=0).stop 2023-09-30T00:23:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a0813fd70 msgr2=0x7f0a08154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a0813fd70 0x7f0a08154610 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0a0000bad0 tx=0x7f0a00009760 comp rx=0 tx=0).stop 2023-09-30T00:23:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/2301839534 shutdown_connections 2023-09-30T00:23:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f09f4098780 0x7f09f409ac30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:00.736 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a0813fd70 0x7f0a08154610 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a08070c50 0x7f0a081409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 1 --2- 172.21.15.94:0/2301839534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a080b7160 0x7f0a081540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/2301839534 >> 172.21.15.94:0/2301839534 conn(0x7f0a0806a090 msgr2=0x7f0a080a8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:23:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/2301839534 shutdown_connections 2023-09-30T00:23:00.737 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:00.735+0000 7f0a0d5fa700 1 -- 172.21.15.94:0/2301839534 wait complete. 2023-09-30T00:23:03.748 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:23:03.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.867+0000 7f39b22e8700 1 Processor -- start 2023-09-30T00:23:03.868 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.867+0000 7f39b22e8700 1 -- start start 2023-09-30T00:23:03.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39b22e8700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f39ac070c50 0x7f39ac0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:03.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39b22e8700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f39ac0b7160 0x7f39ac0a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:03.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39b22e8700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ac0a1310 con 0x7f39ac13fd70 2023-09-30T00:23:03.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39b22e8700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39ac071020 con 0x7f39ac070c50 2023-09-30T00:23:03.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39b22e8700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39ac0b76a0 con 0x7f39ac0b7160 2023-09-30T00:23:03.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39abfff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f39ac070c50 0x7f39ac0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:03.873 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39abfff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f39ac070c50 0x7f39ac0a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56592/0 (socket says 172.21.15.94:56592) 2023-09-30T00:23:03.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39abfff700 1 -- 172.21.15.94:0/313645307 learned_addr learned my addr 172.21.15.94:0/313645307 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:23:03.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39ab7fe700 1 --2- 172.21.15.94:0/313645307 >> v2:172.21.15.121:3300/0 conn(0x7f39ac0b7160 0x7f39ac0a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:03.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39abfff700 1 -- 172.21.15.94:0/313645307 >> v2:172.21.15.121:3300/0 conn(0x7f39ac0b7160 msgr2=0x7f39ac0a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:23:03.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39abfff700 1 --2- 172.21.15.94:0/313645307 >> v2:172.21.15.121:3300/0 conn(0x7f39ac0b7160 0x7f39ac0a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:03.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39abfff700 1 -- 172.21.15.94:0/313645307 >> v1:172.21.15.121:6789/0 conn(0x7f39ac13fd70 legacy=0x7f39ac0b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:23:03.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39abfff700 1 -- 172.21.15.94:0/313645307 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac0b7940 con 0x7f39ac070c50 2023-09-30T00:23:03.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39ab7fe700 1 --2- 172.21.15.94:0/313645307 >> v2:172.21.15.121:3300/0 conn(0x7f39ac0b7160 0x7f39ac0a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:23:03.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39abfff700 1 --2- 172.21.15.94:0/313645307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39ac070c50 0x7f39ac0a0dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f39ac06a880 tx=0x7f399c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b0518628b1786f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:03.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.868+0000 7f39aaffd700 1 -- 172.21.15.94:0/313645307 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c00b8b0 con 0x7f39ac070c50 2023-09-30T00:23:03.874 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.869+0000 7f39aaffd700 1 -- 172.21.15.94:0/313645307 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c00ba10 con 0x7f39ac070c50 2023-09-30T00:23:03.875 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.869+0000 7f39aaffd700 1 -- 172.21.15.94:0/313645307 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c00bdb0 con 0x7f39ac070c50 2023-09-30T00:23:03.875 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.869+0000 7f39b22e8700 1 -- 172.21.15.94:0/313645307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39ac070c50 msgr2=0x7f39ac0a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:03.875 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.869+0000 7f39b22e8700 1 --2- 172.21.15.94:0/313645307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39ac070c50 0x7f39ac0a0dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f39ac06a880 tx=0x7f399c009250 comp rx=0 tx=0).stop 2023-09-30T00:23:03.875 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.869+0000 7f39b22e8700 1 -- 172.21.15.94:0/313645307 shutdown_connections 2023-09-30T00:23:03.875 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.869+0000 7f39b22e8700 1 --2- 172.21.15.94:0/313645307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39ac070c50 0x7f39ac0a0dd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:03.875 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.869+0000 7f39b22e8700 1 --2- 172.21.15.94:0/313645307 >> v2:172.21.15.121:3300/0 conn(0x7f39ac0b7160 0x7f39ac0a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:03.875 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.869+0000 7f39b22e8700 1 -- 172.21.15.94:0/313645307 >> 172.21.15.94:0/313645307 conn(0x7f39ac06a090 msgr2=0x7f39ac0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:23:03.875 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.869+0000 7f39b22e8700 1 -- 172.21.15.94:0/313645307 shutdown_connections 2023-09-30T00:23:03.875 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.869+0000 7f39b22e8700 1 -- 172.21.15.94:0/313645307 wait complete. 2023-09-30T00:23:03.875 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.870+0000 7f39b22e8700 1 Processor -- start 2023-09-30T00:23:03.875 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.870+0000 7f39b22e8700 1 -- start start 2023-09-30T00:23:03.875 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39b22e8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39ac070c50 0x7f39ac0ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39b22e8700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f39ac0b7160 0x7f39ac0a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39b22e8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac13fd70 0x7f39ac14ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39b22e8700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac0abb80 con 0x7f39ac070c50 2023-09-30T00:23:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39b22e8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac144b10 con 0x7f39ac13fd70 2023-09-30T00:23:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39b22e8700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac0ab970 con 0x7f39ac0b7160 2023-09-30T00:23:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39abfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39ac070c50 0x7f39ac0ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39abfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39ac070c50 0x7f39ac0ab430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56606/0 (socket says 172.21.15.94:56606) 2023-09-30T00:23:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39abfff700 1 -- 172.21.15.94:0/1063197230 learned_addr learned my addr 172.21.15.94:0/1063197230 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:23:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39ab7fe700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f39ac0b7160 0x7f39ac0a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39b0885700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac13fd70 0x7f39ac14ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39abfff700 1 -- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f39ac0b7160 msgr2=0x7f39ac0a5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39abfff700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f39ac0b7160 0x7f39ac0a5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39abfff700 1 -- 172.21.15.94:0/1063197230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac13fd70 msgr2=0x7f39ac14ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39abfff700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac13fd70 0x7f39ac14ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39abfff700 1 -- 172.21.15.94:0/1063197230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c0094f0 con 0x7f39ac070c50 2023-09-30T00:23:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39abfff700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39ac070c50 0x7f39ac0ab430 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f399c009290 tx=0x7f399c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f3993fff700 1 -- 172.21.15.94:0/1063197230 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c011d00 con 0x7f39ac070c50 2023-09-30T00:23:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f3993fff700 1 -- 172.21.15.94:0/1063197230 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c020070 con 0x7f39ac070c50 2023-09-30T00:23:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39b22e8700 1 -- 172.21.15.94:0/1063197230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ac1427b0 con 0x7f39ac070c50 2023-09-30T00:23:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f3993fff700 1 -- 172.21.15.94:0/1063197230 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c01b6a0 con 0x7f39ac070c50 2023-09-30T00:23:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.871+0000 7f39b22e8700 1 -- 172.21.15.94:0/1063197230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ac0aacd0 con 0x7f39ac070c50 2023-09-30T00:23:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.872+0000 7f39b22e8700 1 -- 172.21.15.94:0/1063197230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ac13e430 con 0x7f39ac070c50 2023-09-30T00:23:03.878 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.873+0000 7f3993fff700 1 -- 172.21.15.94:0/1063197230 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f399c022030 con 0x7f39ac070c50 2023-09-30T00:23:03.878 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.873+0000 7f3993fff700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:23:03.878 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.874+0000 7f3993fff700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:23:03.878 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.874+0000 7f3993fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:23:03.879 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.874+0000 7f3993fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:23:03.879 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.874+0000 7f3993fff700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3998098780 0x7f399809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:03.879 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.874+0000 7f39ab7fe700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3998098780 0x7f399809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:03.879 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.874+0000 7f3993fff700 1 -- 172.21.15.94:0/1063197230 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f399c0c20e0 con 0x7f39ac070c50 2023-09-30T00:23:03.879 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.874+0000 7f39ab7fe700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3998098780 0x7f399809ac30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0b47d0 tx=0x7f39a0006f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:03.879 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:03.878+0000 7f3993fff700 1 -- 172.21.15.94:0/1063197230 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f399c08f7b0 con 0x7f39ac070c50 2023-09-30T00:23:04.105 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.104+0000 7f39b22e8700 1 -- 172.21.15.94:0/1063197230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f39ac0a1310 con 0x7f39ac070c50 2023-09-30T00:23:04.105 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.105+0000 7f3993fff700 1 -- 172.21.15.94:0/1063197230 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f39ac0a1310 con 0x7f39ac070c50 2023-09-30T00:23:04.105 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:23:04.105 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:23:04.107 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.106+0000 7f39b22e8700 10 mgrc shutdown 2023-09-30T00:23:04.107 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.106+0000 7f39b22e8700 1 -- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3998098780 msgr2=0x7f399809ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:04.107 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.106+0000 7f39b22e8700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3998098780 0x7f399809ac30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0b47d0 tx=0x7f39a0006f90 comp rx=0 tx=0).stop 2023-09-30T00:23:04.107 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.106+0000 7f39b22e8700 1 -- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39ac070c50 msgr2=0x7f39ac0ab430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:04.108 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.106+0000 7f39b22e8700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39ac070c50 0x7f39ac0ab430 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f399c009290 tx=0x7f399c016000 comp rx=0 tx=0).stop 2023-09-30T00:23:04.108 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.106+0000 7f39b22e8700 1 -- 172.21.15.94:0/1063197230 shutdown_connections 2023-09-30T00:23:04.108 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.106+0000 7f39b22e8700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f3998098780 0x7f399809ac30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:04.108 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.106+0000 7f39b22e8700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac13fd70 0x7f39ac14ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:04.108 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.106+0000 7f39b22e8700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39ac070c50 0x7f39ac0ab430 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:04.108 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.106+0000 7f39b22e8700 1 --2- 172.21.15.94:0/1063197230 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f39ac0b7160 0x7f39ac0a5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:04.108 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.106+0000 7f39b22e8700 1 -- 172.21.15.94:0/1063197230 >> 172.21.15.94:0/1063197230 conn(0x7f39ac06a090 msgr2=0x7f39ac13d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:23:04.108 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.107+0000 7f39b22e8700 1 -- 172.21.15.94:0/1063197230 shutdown_connections 2023-09-30T00:23:04.108 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:04.107+0000 7f39b22e8700 1 -- 172.21.15.94:0/1063197230 wait complete. 2023-09-30T00:23:07.120 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:23:07.244 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.244+0000 7fcee6cf2700 1 Processor -- start 2023-09-30T00:23:07.244 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.244+0000 7fcee6cf2700 1 -- start start 2023-09-30T00:23:07.247 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.244+0000 7fcee6cf2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee006aaa0 0x7fcee00b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.247 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.244+0000 7fcee6cf2700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcee00a3ff0 0x7fcee00b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.247 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.244+0000 7fcee6cf2700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcee009ca70 0x7fcee009ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.247 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.244+0000 7fcee6cf2700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee006ae70 con 0x7fcee009ca70 2023-09-30T00:23:07.247 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.244+0000 7fcee6cf2700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee00b40e0 con 0x7fcee00a3ff0 2023-09-30T00:23:07.247 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.244+0000 7fcee6cf2700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00b48c0 con 0x7fcee006aaa0 2023-09-30T00:23:07.247 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.244+0000 7fcee4a8e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee006aaa0 0x7fcee00b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.244+0000 7fcee528f700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcee009ca70 0x7fcee009ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fcee528f700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcee009ca70 0x7fcee009ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41974/0 (socket says 172.21.15.94:41974) 2023-09-30T00:23:07.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fcee4a8e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee006aaa0 0x7fcee00b3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:35316/0 (socket says 172.21.15.94:35316) 2023-09-30T00:23:07.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fcee528f700 1 -- 172.21.15.94:0/2669449546 learned_addr learned my addr 172.21.15.94:0/2669449546 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:23:07.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fced7fff700 1 --2- 172.21.15.94:0/2669449546 >> v2:172.21.15.121:3300/0 conn(0x7fcee00a3ff0 0x7fcee00b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fcee528f700 1 -- 172.21.15.94:0/2669449546 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee006aaa0 msgr2=0x7fcee00b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:23:07.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fcee528f700 1 --2- 172.21.15.94:0/2669449546 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee006aaa0 0x7fcee00b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fcee528f700 1 -- 172.21.15.94:0/2669449546 >> v2:172.21.15.121:3300/0 conn(0x7fcee00a3ff0 msgr2=0x7fcee00b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:23:07.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fcee528f700 1 --2- 172.21.15.94:0/2669449546 >> v2:172.21.15.121:3300/0 conn(0x7fcee00a3ff0 0x7fcee00b24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fcee528f700 1 -- 172.21.15.94:0/2669449546 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee00a45c0 con 0x7fcee009ca70 2023-09-30T00:23:07.248 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fcee4a8e700 1 --2- 172.21.15.94:0/2669449546 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee006aaa0 0x7fcee00b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:23:07.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fcee528f700 1 --2- 172.21.15.94:0/2669449546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee009ca70 0x7fcee009ceb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcee006b080 tx=0x7fced800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1679782ea7cea18 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:07.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fced77fe700 1 -- 172.21.15.94:0/2669449546 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8010070 con 0x7fcee009ca70 2023-09-30T00:23:07.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fced77fe700 1 -- 172.21.15.94:0/2669449546 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced800dde0 con 0x7fcee009ca70 2023-09-30T00:23:07.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.245+0000 7fced77fe700 1 -- 172.21.15.94:0/2669449546 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced800b320 con 0x7fcee009ca70 2023-09-30T00:23:07.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.246+0000 7fcee6cf2700 1 -- 172.21.15.94:0/2669449546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee009ca70 msgr2=0x7fcee009ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.246+0000 7fcee6cf2700 1 --2- 172.21.15.94:0/2669449546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee009ca70 0x7fcee009ceb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcee006b080 tx=0x7fced800ba50 comp rx=0 tx=0).stop 2023-09-30T00:23:07.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.246+0000 7fcee6cf2700 1 -- 172.21.15.94:0/2669449546 shutdown_connections 2023-09-30T00:23:07.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.246+0000 7fcee6cf2700 1 --2- 172.21.15.94:0/2669449546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee009ca70 0x7fcee009ceb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.246+0000 7fcee6cf2700 1 --2- 172.21.15.94:0/2669449546 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee006aaa0 0x7fcee00b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.246+0000 7fcee6cf2700 1 --2- 172.21.15.94:0/2669449546 >> v2:172.21.15.121:3300/0 conn(0x7fcee00a3ff0 0x7fcee00b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.246+0000 7fcee6cf2700 1 -- 172.21.15.94:0/2669449546 >> 172.21.15.94:0/2669449546 conn(0x7fcee0068730 msgr2=0x7fcee0068b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:23:07.249 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.246+0000 7fcee6cf2700 1 -- 172.21.15.94:0/2669449546 shutdown_connections 2023-09-30T00:23:07.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.246+0000 7fcee6cf2700 1 -- 172.21.15.94:0/2669449546 wait complete. 2023-09-30T00:23:07.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.247+0000 7fcee6cf2700 1 Processor -- start 2023-09-30T00:23:07.250 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.247+0000 7fcee6cf2700 1 -- start start 2023-09-30T00:23:07.252 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.247+0000 7fcee6cf2700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee006aaa0 0x7fcee0140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.252 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.247+0000 7fcee6cf2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee009ca70 0x7fcee009f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.252 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.247+0000 7fcee6cf2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee00a3ff0 0x7fcee00a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.252 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.247+0000 7fcee6cf2700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee009fea0 con 0x7fcee006aaa0 2023-09-30T00:23:07.252 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.247+0000 7fcee6cf2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee009c650 con 0x7fcee00a3ff0 2023-09-30T00:23:07.252 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.247+0000 7fcee6cf2700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00a1ac0 con 0x7fcee009ca70 2023-09-30T00:23:07.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.247+0000 7fcee4a8e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee006aaa0 0x7fcee0140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.247+0000 7fced7fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee009ca70 0x7fcee009f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.247+0000 7fcee4a8e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee006aaa0 0x7fcee0140b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41990/0 (socket says 172.21.15.94:41990) 2023-09-30T00:23:07.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.247+0000 7fcee4a8e700 1 -- 172.21.15.94:0/4197547441 learned_addr learned my addr 172.21.15.94:0/4197547441 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:23:07.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.248+0000 7fcee528f700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee00a3ff0 0x7fcee00a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.248+0000 7fcee4a8e700 1 -- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee009ca70 msgr2=0x7fcee009f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.248+0000 7fcee4a8e700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee009ca70 0x7fcee009f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.248+0000 7fcee4a8e700 1 -- 172.21.15.94:0/4197547441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee00a3ff0 msgr2=0x7fcee00a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.248+0000 7fcee4a8e700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee00a3ff0 0x7fcee00a1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.248+0000 7fcee4a8e700 1 -- 172.21.15.94:0/4197547441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc009420 con 0x7fcee006aaa0 2023-09-30T00:23:07.253 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.248+0000 7fcee4a8e700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee006aaa0 0x7fcee0140b50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcedc00ec80 tx=0x7fcedc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:07.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.248+0000 7fced4ff9700 1 -- 172.21.15.94:0/4197547441 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc00c980 con 0x7fcee006aaa0 2023-09-30T00:23:07.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.248+0000 7fced4ff9700 1 -- 172.21.15.94:0/4197547441 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcedc00cae0 con 0x7fcee006aaa0 2023-09-30T00:23:07.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.248+0000 7fcee6cf2700 1 -- 172.21.15.94:0/4197547441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced800bcf0 con 0x7fcee006aaa0 2023-09-30T00:23:07.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.248+0000 7fced4ff9700 1 -- 172.21.15.94:0/4197547441 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc00ce60 con 0x7fcee006aaa0 2023-09-30T00:23:07.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.248+0000 7fcee6cf2700 1 -- 172.21.15.94:0/4197547441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee01411f0 con 0x7fcee006aaa0 2023-09-30T00:23:07.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.249+0000 7fcee6cf2700 1 -- 172.21.15.94:0/4197547441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee006fbd0 con 0x7fcee006aaa0 2023-09-30T00:23:07.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.250+0000 7fced4ff9700 1 -- 172.21.15.94:0/4197547441 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fcedc013680 con 0x7fcee006aaa0 2023-09-30T00:23:07.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.250+0000 7fced4ff9700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:23:07.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.250+0000 7fced4ff9700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:23:07.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.250+0000 7fced4ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:23:07.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.250+0000 7fced4ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:23:07.254 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.250+0000 7fced4ff9700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcecc098780 0x7fcecc09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.250+0000 7fced7fff700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcecc098780 0x7fcecc09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.250+0000 7fced4ff9700 1 -- 172.21.15.94:0/4197547441 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fcedc0c0c30 con 0x7fcee006aaa0 2023-09-30T00:23:07.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.251+0000 7fced7fff700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcecc098780 0x7fcecc09ac30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcee014eed0 tx=0x7fced0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:07.255 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.255+0000 7fced4ff9700 1 -- 172.21.15.94:0/4197547441 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcedc091d00 con 0x7fcee006aaa0 2023-09-30T00:23:07.479 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.478+0000 7fcee6cf2700 1 -- 172.21.15.94:0/4197547441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcee00b4360 con 0x7fcee006aaa0 2023-09-30T00:23:07.479 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.479+0000 7fced4ff9700 1 -- 172.21.15.94:0/4197547441 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcedc021100 con 0x7fcee006aaa0 2023-09-30T00:23:07.479 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:23:07.479 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:23:07.481 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 10 mgrc shutdown 2023-09-30T00:23:07.481 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 1 -- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcecc098780 msgr2=0x7fcecc09ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcecc098780 0x7fcecc09ac30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcee014eed0 tx=0x7fced0008040 comp rx=0 tx=0).stop 2023-09-30T00:23:07.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 1 -- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee006aaa0 msgr2=0x7fcee0140b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee006aaa0 0x7fcee0140b50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcedc00ec80 tx=0x7fcedc0091a0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 1 -- 172.21.15.94:0/4197547441 shutdown_connections 2023-09-30T00:23:07.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fcecc098780 0x7fcecc09ac30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee00a3ff0 0x7fcee00a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee006aaa0 0x7fcee0140b50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 1 --2- 172.21.15.94:0/4197547441 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee009ca70 0x7fcee009f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 1 -- 172.21.15.94:0/4197547441 >> 172.21.15.94:0/4197547441 conn(0x7fcee0068730 msgr2=0x7fcee0068b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:23:07.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 1 -- 172.21.15.94:0/4197547441 shutdown_connections 2023-09-30T00:23:07.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.481+0000 7fcee6cf2700 1 -- 172.21.15.94:0/4197547441 wait complete. 2023-09-30T00:23:07.493 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T00:23:07.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:23:07.606 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.605+0000 7fd1de1be700 1 Processor -- start 2023-09-30T00:23:07.606 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.605+0000 7fd1de1be700 1 -- start start 2023-09-30T00:23:07.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1de1be700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d80a46c0 0x7fd1d8068c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1de1be700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd1d806a460 0x7fd1d80b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1de1be700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d8069760 con 0x7fd1d80691b0 2023-09-30T00:23:07.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1de1be700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d80a57f0 con 0x7fd1d806a460 2023-09-30T00:23:07.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1de1be700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80a4a90 con 0x7fd1d80a46c0 2023-09-30T00:23:07.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1d77fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d80a46c0 0x7fd1d8068c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1d77fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d80a46c0 0x7fd1d8068c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:35324/0 (socket says 172.21.15.94:35324) 2023-09-30T00:23:07.611 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1d77fe700 1 -- 172.21.15.94:0/544883794 learned_addr learned my addr 172.21.15.94:0/544883794 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:23:07.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1d77fe700 1 -- 172.21.15.94:0/544883794 >> v2:172.21.15.121:3300/0 conn(0x7fd1d806a460 msgr2=0x7fd1d80b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:23:07.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1d6ffd700 1 --2- 172.21.15.94:0/544883794 >> v2:172.21.15.121:3300/0 conn(0x7fd1d806a460 0x7fd1d80b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1d77fe700 1 --2- 172.21.15.94:0/544883794 >> v2:172.21.15.121:3300/0 conn(0x7fd1d806a460 0x7fd1d80b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1d77fe700 1 -- 172.21.15.94:0/544883794 >> v1:172.21.15.121:6789/0 conn(0x7fd1d80691b0 legacy=0x7fd1d8143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:23:07.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1d77fe700 1 -- 172.21.15.94:0/544883794 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8069c80 con 0x7fd1d80a46c0 2023-09-30T00:23:07.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1d6ffd700 1 --2- 172.21.15.94:0/544883794 >> v2:172.21.15.121:3300/0 conn(0x7fd1d806a460 0x7fd1d80b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:23:07.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.606+0000 7fd1d77fe700 1 --2- 172.21.15.94:0/544883794 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d80a46c0 0x7fd1d8068c70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009f20 tx=0x7fd1c8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=798f258a63987e78 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:07.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.607+0000 7fd1d67fc700 1 -- 172.21.15.94:0/544883794 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c800e070 con 0x7fd1d80a46c0 2023-09-30T00:23:07.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.607+0000 7fd1d67fc700 1 -- 172.21.15.94:0/544883794 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c800ba40 con 0x7fd1d80a46c0 2023-09-30T00:23:07.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.607+0000 7fd1d67fc700 1 -- 172.21.15.94:0/544883794 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80092e0 con 0x7fd1d80a46c0 2023-09-30T00:23:07.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.607+0000 7fd1de1be700 1 -- 172.21.15.94:0/544883794 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d80a46c0 msgr2=0x7fd1d8068c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.612 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.607+0000 7fd1de1be700 1 --2- 172.21.15.94:0/544883794 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d80a46c0 0x7fd1d8068c70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009f20 tx=0x7fd1c8009840 comp rx=0 tx=0).stop 2023-09-30T00:23:07.613 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.607+0000 7fd1de1be700 1 -- 172.21.15.94:0/544883794 shutdown_connections 2023-09-30T00:23:07.613 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.607+0000 7fd1de1be700 1 --2- 172.21.15.94:0/544883794 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d80a46c0 0x7fd1d8068c70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.613 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.607+0000 7fd1de1be700 1 --2- 172.21.15.94:0/544883794 >> v2:172.21.15.121:3300/0 conn(0x7fd1d806a460 0x7fd1d80b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.613 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.607+0000 7fd1de1be700 1 -- 172.21.15.94:0/544883794 >> 172.21.15.94:0/544883794 conn(0x7fd1d806cbf0 msgr2=0x7fd1d806cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:23:07.613 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.607+0000 7fd1de1be700 1 -- 172.21.15.94:0/544883794 shutdown_connections 2023-09-30T00:23:07.613 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.607+0000 7fd1de1be700 1 -- 172.21.15.94:0/544883794 wait complete. 2023-09-30T00:23:07.613 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.608+0000 7fd1de1be700 1 Processor -- start 2023-09-30T00:23:07.613 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.608+0000 7fd1de1be700 1 -- start start 2023-09-30T00:23:07.613 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1de1be700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1d80691b0 0x7fd1d811cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.613 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1de1be700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d806a460 0x7fd1d813f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.614 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1de1be700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1d80a46c0 0x7fd1d813ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.614 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1de1be700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80b1020 con 0x7fd1d80a46c0 2023-09-30T00:23:07.614 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1de1be700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80af0c0 con 0x7fd1d80691b0 2023-09-30T00:23:07.614 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1de1be700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80b1800 con 0x7fd1d806a460 2023-09-30T00:23:07.614 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1d6ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d806a460 0x7fd1d813f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.614 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1d6ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d806a460 0x7fd1d813f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:35340/0 (socket says 172.21.15.94:35340) 2023-09-30T00:23:07.614 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1d6ffd700 1 -- 172.21.15.94:0/409289299 learned_addr learned my addr 172.21.15.94:0/409289299 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:23:07.614 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1d7fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1d80a46c0 0x7fd1d813ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.614 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1d77fe700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1d80691b0 0x7fd1d811cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1d6ffd700 1 -- 172.21.15.94:0/409289299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1d80691b0 msgr2=0x7fd1d811cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1d6ffd700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1d80691b0 0x7fd1d811cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1d6ffd700 1 -- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1d80a46c0 msgr2=0x7fd1d813ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1d6ffd700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1d80a46c0 0x7fd1d813ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1d6ffd700 1 -- 172.21.15.94:0/409289299 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c8009ae0 con 0x7fd1d806a460 2023-09-30T00:23:07.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1d7fff700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1d80a46c0 0x7fd1d813ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:23:07.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1d6ffd700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d806a460 0x7fd1d813f710 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00c830 tx=0x7fd1cc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:07.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1bf7fe700 1 -- 172.21.15.94:0/409289299 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc011600 con 0x7fd1d806a460 2023-09-30T00:23:07.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1bf7fe700 1 -- 172.21.15.94:0/409289299 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1cc011760 con 0x7fd1d806a460 2023-09-30T00:23:07.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1de1be700 1 -- 172.21.15.94:0/409289299 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d80b1a30 con 0x7fd1d806a460 2023-09-30T00:23:07.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.609+0000 7fd1bf7fe700 1 -- 172.21.15.94:0/409289299 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc011b20 con 0x7fd1d806a460 2023-09-30T00:23:07.615 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.610+0000 7fd1de1be700 1 -- 172.21.15.94:0/409289299 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d81406b0 con 0x7fd1d806a460 2023-09-30T00:23:07.616 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.610+0000 7fd1de1be700 1 -- 172.21.15.94:0/409289299 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d8140960 con 0x7fd1d806a460 2023-09-30T00:23:07.617 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.611+0000 7fd1bf7fe700 1 -- 172.21.15.94:0/409289299 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7fd1cc018360 con 0x7fd1d806a460 2023-09-30T00:23:07.617 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.611+0000 7fd1bf7fe700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:23:07.617 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.612+0000 7fd1bf7fe700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:23:07.617 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.612+0000 7fd1bf7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:23:07.617 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.612+0000 7fd1bf7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:23:07.617 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.612+0000 7fd1bf7fe700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd1c40987b0 0x7fd1c409ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.617 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.612+0000 7fd1d77fe700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd1c40987b0 0x7fd1c409ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.617 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.612+0000 7fd1bf7fe700 1 -- 172.21.15.94:0/409289299 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7fd1cc0c0380 con 0x7fd1d806a460 2023-09-30T00:23:07.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.612+0000 7fd1d77fe700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd1c40987b0 0x7fd1c409ac60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009880 tx=0x7fd1c8017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:07.618 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.616+0000 7fd1bf7fe700 1 -- 172.21.15.94:0/409289299 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd1cc08da50 con 0x7fd1d806a460 2023-09-30T00:23:07.798 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.798+0000 7fd1de1be700 1 -- 172.21.15.94:0/409289299 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1d806fbd0 con 0x7fd1d806a460 2023-09-30T00:23:07.799 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.798+0000 7fd1bf7fe700 1 -- 172.21.15.94:0/409289299 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+12384 (secure 0 0 0) 0x7fd1cc091070 con 0x7fd1d806a460 2023-09-30T00:23:07.799 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-09-30T00:23:07.800 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":21,"fsid":"3c891173-587e-4c6f-9c6a-04ec750eedad","created":"2023-09-30T00:17:34.347592+0000","modified":"2023-09-30T00:18:41.183955+0000","last_up_change":"2023-09-30T00:17:42.879328+0000","last_in_change":"2023-09-30T00:17:35.583427+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T00:17:43.481613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T00:17:45.892643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-09-30T00:17:50.220028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-09-30T00:17:50.911066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c508b402-f8ce-4461-823b-9e35fcd8d40f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":84364},{"type":"v1","addr":"172.21.15.94:6801","nonce":84364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":84364},{"type":"v1","addr":"172.21.15.94:6803","nonce":84364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":84364},{"type":"v1","addr":"172.21.15.94:6807","nonce":84364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":84364},{"type":"v1","addr":"172.21.15.94:6805","nonce":84364}]},"public_addr":"172.21.15.94:6801/84364","cluster_addr":"172.21.15.94:6803/84364","heartbeat_back_addr":"172.21.15.94:6807/84364","heartbeat_front_addr":"172.21.15.94:6805/84364","state":["exists","up"]},{"osd":1,"uuid":"e96ecdf0-bb38-4eb1-be99-54f813c12888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":86850},{"type":"v1","addr":"172.21.15.121:6817","nonce":86850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":86850},{"type":"v1","addr":"172.21.15.121:6819","nonce":86850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":86850},{"type":"v1","addr":"172.21.15.121:6823","nonce":86850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":86850},{"type":"v1","addr":"172.21.15.121:6821","nonce":86850}]},"public_addr":"172.21.15.121:6817/86850","cluster_addr":"172.21.15.121:6819/86850","heartbeat_back_addr":"172.21.15.121:6823/86850","heartbeat_front_addr":"172.21.15.121:6821/86850","state":["exists","up"]},{"osd":2,"uuid":"68700ae4-764c-4dff-9cf2-5f33132cd0d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":86820},{"type":"v1","addr":"172.21.15.121:6801","nonce":86820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":86820},{"type":"v1","addr":"172.21.15.121:6803","nonce":86820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":86820},{"type":"v1","addr":"172.21.15.121:6810","nonce":86820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":86820},{"type":"v1","addr":"172.21.15.121:6806","nonce":86820}]},"public_addr":"172.21.15.121:6801/86820","cluster_addr":"172.21.15.121:6803/86820","heartbeat_back_addr":"172.21.15.121:6810/86820","heartbeat_front_addr":"172.21.15.121:6806/86820","state":["exists","up"]},{"osd":3,"uuid":"54f17ec8-2561-42e8-bbd7-262130ffdac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6805","nonce":86821},{"type":"v1","addr":"172.21.15.121:6807","nonce":86821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6809","nonce":86821},{"type":"v1","addr":"172.21.15.121:6811","nonce":86821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":86821},{"type":"v1","addr":"172.21.15.121:6815","nonce":86821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":86821},{"type":"v1","addr":"172.21.15.121:6813","nonce":86821}]},"public_addr":"172.21.15.121:6807/86821","cluster_addr":"172.21.15.121:6811/86821","heartbeat_back_addr":"172.21.15.121:6815/86821","heartbeat_front_addr":"172.21.15.121:6813/86821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:39.951865+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.769149+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.738994+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:17:40.748197+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.6","mappings":[{"from":3,"to":2}]},{"pgid":"4.a","mappings":[{"from":3,"to":0}]},{"pgid":"4.f","mappings":[{"from":3,"to":2}]},{"pgid":"4.16","mappings":[{"from":3,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:23:07.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 10 mgrc shutdown 2023-09-30T00:23:07.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 1 -- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd1c40987b0 msgr2=0x7fd1c409ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd1c40987b0 0x7fd1c409ac60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009880 tx=0x7fd1c8017000 comp rx=0 tx=0).stop 2023-09-30T00:23:07.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 1 -- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d806a460 msgr2=0x7fd1d813f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d806a460 0x7fd1d813f710 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00c830 tx=0x7fd1cc008040 comp rx=0 tx=0).stop 2023-09-30T00:23:07.802 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 1 -- 172.21.15.94:0/409289299 shutdown_connections 2023-09-30T00:23:07.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7fd1c40987b0 0x7fd1c409ac60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1d80a46c0 0x7fd1d813ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1d80691b0 0x7fd1d811cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 1 --2- 172.21.15.94:0/409289299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1d806a460 0x7fd1d813f710 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 1 -- 172.21.15.94:0/409289299 >> 172.21.15.94:0/409289299 conn(0x7fd1d806cbf0 msgr2=0x7fd1d8140fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:23:07.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 1 -- 172.21.15.94:0/409289299 shutdown_connections 2023-09-30T00:23:07.803 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.801+0000 7fd1de1be700 1 -- 172.21.15.94:0/409289299 wait complete. 2023-09-30T00:23:07.814 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T00:23:07.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.926+0000 7f0c1424c700 1 Processor -- start 2023-09-30T00:23:07.927 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.926+0000 7f0c1424c700 1 -- start start 2023-09-30T00:23:07.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.927+0000 7f0c1424c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c0c06aaa0 0x7f0c0c0a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.929 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.927+0000 7f0c1424c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0c0c0b25c0 0x7f0c0c0b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.927+0000 7f0c1424c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0c06ae70 con 0x7f0c0c143c30 2023-09-30T00:23:07.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.927+0000 7f0c1424c700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0c1448c0 con 0x7f0c0c0b25c0 2023-09-30T00:23:07.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.927+0000 7f0c1424c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c136db0 con 0x7f0c0c06aaa0 2023-09-30T00:23:07.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.927+0000 7f0c117e7700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0c0c0b25c0 0x7f0c0c0b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.927+0000 7f0c117e7700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0c0c0b25c0 0x7f0c0c0b29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42004/0 (socket says 172.21.15.94:42004) 2023-09-30T00:23:07.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.927+0000 7f0c127e9700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0c0c143c30 0x7f0c0c144090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35180/0 (socket says 172.21.15.94:35180) 2023-09-30T00:23:07.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.927+0000 7f0c117e7700 1 -- 172.21.15.94:0/91101541 learned_addr learned my addr 172.21.15.94:0/91101541 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:23:07.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.927+0000 7f0c11fe8700 1 --2- 172.21.15.94:0/91101541 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c0c06aaa0 0x7f0c0c0a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c117e7700 1 -- 172.21.15.94:0/91101541 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c0c06aaa0 msgr2=0x7f0c0c0a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:23:07.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c117e7700 1 --2- 172.21.15.94:0/91101541 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c0c06aaa0 0x7f0c0c0a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.930 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c117e7700 1 -- 172.21.15.94:0/91101541 >> v1:172.21.15.94:6789/0 conn(0x7f0c0c143c30 legacy=0x7f0c0c144090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c10fe6700 1 -- 172.21.15.94:0/91101541 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342776068 0 0) 0x7f0c0c06ae70 con 0x7f0c0c143c30 2023-09-30T00:23:07.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c117e7700 1 -- 172.21.15.94:0/91101541 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c0b3490 con 0x7f0c0c0b25c0 2023-09-30T00:23:07.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c117e7700 1 --2- 172.21.15.94:0/91101541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c0c0b25c0 0x7f0c0c0b29d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0099b0 tx=0x7f0bfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e09cfbb7423c1a2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:07.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c10fe6700 1 -- 172.21.15.94:0/91101541 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc009ea0 con 0x7f0c0c0b25c0 2023-09-30T00:23:07.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c10fe6700 1 -- 172.21.15.94:0/91101541 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc00b860 con 0x7f0c0c0b25c0 2023-09-30T00:23:07.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c10fe6700 1 -- 172.21.15.94:0/91101541 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc00bc00 con 0x7f0c0c0b25c0 2023-09-30T00:23:07.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c1424c700 1 -- 172.21.15.94:0/91101541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c0c0b25c0 msgr2=0x7f0c0c0b29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c1424c700 1 --2- 172.21.15.94:0/91101541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c0c0b25c0 0x7f0c0c0b29d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0099b0 tx=0x7f0bfc009250 comp rx=0 tx=0).stop 2023-09-30T00:23:07.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c1424c700 1 -- 172.21.15.94:0/91101541 shutdown_connections 2023-09-30T00:23:07.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c1424c700 1 --2- 172.21.15.94:0/91101541 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c0c06aaa0 0x7f0c0c0a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c1424c700 1 --2- 172.21.15.94:0/91101541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c0c0b25c0 0x7f0c0c0b29d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.931 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.928+0000 7f0c1424c700 1 -- 172.21.15.94:0/91101541 >> 172.21.15.94:0/91101541 conn(0x7f0c0c068730 msgr2=0x7f0c0c068b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:23:07.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.929+0000 7f0c1424c700 1 -- 172.21.15.94:0/91101541 shutdown_connections 2023-09-30T00:23:07.932 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.929+0000 7f0c1424c700 1 -- 172.21.15.94:0/91101541 wait complete. 2023-09-30T00:23:07.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.929+0000 7f0c1424c700 1 Processor -- start 2023-09-30T00:23:07.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c1424c700 1 -- start start 2023-09-30T00:23:07.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c1424c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c0c06aaa0 0x7f0c0c0a2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.934 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c1424c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c0c0b25c0 0x7f0c0c0a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c1424c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c0c143c30 0x7f0c0c14eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c1424c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0a1700 con 0x7f0c0c06aaa0 2023-09-30T00:23:07.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c1424c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0a19c0 con 0x7f0c0c0b25c0 2023-09-30T00:23:07.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c1424c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c1510a0 con 0x7f0c0c143c30 2023-09-30T00:23:07.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c11fe8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c0c06aaa0 0x7f0c0c0a2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c11fe8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c0c06aaa0 0x7f0c0c0a2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42012/0 (socket says 172.21.15.94:42012) 2023-09-30T00:23:07.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c127e9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c0c143c30 0x7f0c0c14eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c11fe8700 1 -- 172.21.15.94:0/904763345 learned_addr learned my addr 172.21.15.94:0/904763345 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-09-30T00:23:07.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c117e7700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c0c0b25c0 0x7f0c0c0a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c11fe8700 1 -- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c0c143c30 msgr2=0x7f0c0c14eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.935 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c11fe8700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c0c143c30 0x7f0c0c14eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c11fe8700 1 -- 172.21.15.94:0/904763345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c0c0b25c0 msgr2=0x7f0c0c0a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c11fe8700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c0c0b25c0 0x7f0c0c0a09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c11fe8700 1 -- 172.21.15.94:0/904763345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc0094f0 con 0x7f0c0c06aaa0 2023-09-30T00:23:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.930+0000 7f0c127e9700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c0c143c30 0x7f0c0c14eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:23:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.931+0000 7f0c11fe8700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c0c06aaa0 0x7f0c0c0a2a90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009e50 tx=0x7f0c0800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.931+0000 7f0c01ffb700 1 -- 172.21.15.94:0/904763345 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0800cb50 con 0x7f0c0c06aaa0 2023-09-30T00:23:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.931+0000 7f0c01ffb700 1 -- 172.21.15.94:0/904763345 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0800f040 con 0x7f0c0c06aaa0 2023-09-30T00:23:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.931+0000 7f0c1424c700 1 -- 172.21.15.94:0/904763345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0c09cb70 con 0x7f0c0c06aaa0 2023-09-30T00:23:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.931+0000 7f0c01ffb700 1 -- 172.21.15.94:0/904763345 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c08014480 con 0x7f0c0c06aaa0 2023-09-30T00:23:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.931+0000 7f0c1424c700 1 -- 172.21.15.94:0/904763345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0c0a38b0 con 0x7f0c0c06aaa0 2023-09-30T00:23:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.932+0000 7f0c1424c700 1 -- 172.21.15.94:0/904763345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0c071630 con 0x7f0c0c06aaa0 2023-09-30T00:23:07.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.933+0000 7f0c01ffb700 1 -- 172.21.15.94:0/904763345 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 126939+0+0 (secure 0 0 0) 0x7f0c080147f0 con 0x7f0c0c06aaa0 2023-09-30T00:23:07.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.933+0000 7f0c01ffb700 20 mgrc handle_mgr_map mgrmap(e 5) v1 2023-09-30T00:23:07.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.933+0000 7f0c01ffb700 4 mgrc handle_mgr_map Got map version 5 2023-09-30T00:23:07.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.933+0000 7f0c01ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:23:07.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.933+0000 7f0c01ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] 2023-09-30T00:23:07.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.933+0000 7f0c01ffb700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f0bf0098780 0x7f0bf009ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:23:07.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.934+0000 7f0c117e7700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f0bf0098780 0x7f0bf009ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:23:07.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.934+0000 7f0c01ffb700 1 -- 172.21.15.94:0/904763345 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4446+0+0 (secure 0 0 0) 0x7f0c080c1620 con 0x7f0c0c06aaa0 2023-09-30T00:23:07.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.934+0000 7f0c117e7700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f0bf0098780 0x7f0bf009ac30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0099b0 tx=0x7f0bfc017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:23:07.938 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:07.937+0000 7f0c01ffb700 1 -- 172.21.15.94:0/904763345 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c0808ebc0 con 0x7f0c0c06aaa0 2023-09-30T00:23:08.119 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.119+0000 7f0c1424c700 1 -- 172.21.15.94:0/904763345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f0c0c0a61f0 con 0x7f0c0c06aaa0 2023-09-30T00:23:08.123 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.123+0000 7f0c01ffb700 1 -- 172.21.15.94:0/904763345 <== mon.0 v2:172.21.15.94:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f0c0808ebc0 con 0x7f0c0c06aaa0 2023-09-30T00:23:08.123 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.123+0000 7f0c01ffb700 1 -- 172.21.15.94:0/904763345 <== mon.0 v2:172.21.15.94:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) v1 ==== 123+0+0 (secure 0 0 0) 0x7f0c080921e0 con 0x7f0c0c06aaa0 2023-09-30T00:23:08.126 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.125+0000 7f0c1424c700 10 mgrc shutdown 2023-09-30T00:23:08.126 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.125+0000 7f0c1424c700 1 -- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f0bf0098780 msgr2=0x7f0bf009ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:08.126 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.125+0000 7f0c1424c700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f0bf0098780 0x7f0bf009ac30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0099b0 tx=0x7f0bfc017000 comp rx=0 tx=0).stop 2023-09-30T00:23:08.126 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.125+0000 7f0c1424c700 1 -- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c0c06aaa0 msgr2=0x7f0c0c0a2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:23:08.127 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.125+0000 7f0c1424c700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c0c06aaa0 0x7f0c0c0a2a90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009e50 tx=0x7f0c0800b040 comp rx=0 tx=0).stop 2023-09-30T00:23:08.127 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.126+0000 7f0c1424c700 1 -- 172.21.15.94:0/904763345 shutdown_connections 2023-09-30T00:23:08.127 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.126+0000 7f0c1424c700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:6808/84242,v1:172.21.15.94:6809/84242] conn(0x7f0bf0098780 0x7f0bf009ac30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:08.127 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.126+0000 7f0c1424c700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c0c143c30 0x7f0c0c14eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:08.127 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.126+0000 7f0c1424c700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c0c06aaa0 0x7f0c0c0a2a90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:08.127 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.126+0000 7f0c1424c700 1 --2- 172.21.15.94:0/904763345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c0c0b25c0 0x7f0c0c0a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:23:08.127 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.126+0000 7f0c1424c700 1 -- 172.21.15.94:0/904763345 >> 172.21.15.94:0/904763345 conn(0x7f0c0c068730 msgr2=0x7f0c0c06e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:23:08.127 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.126+0000 7f0c1424c700 1 -- 172.21.15.94:0/904763345 shutdown_connections 2023-09-30T00:23:08.127 INFO:teuthology.orchestra.run.smithi094.stderr:2023-09-30T00:23:08.126+0000 7f0c1424c700 1 -- 172.21.15.94:0/904763345 wait complete. 2023-09-30T00:23:08.138 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:23:08.139 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T00:23:08.140 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-09-30T00:23:08.140 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.148 INFO:tasks.ceph.mds.a:Stopped 2023-09-30T00:23:08.148 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-09-30T00:23:08.148 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.197 INFO:tasks.ceph.mds.c:Stopped 2023-09-30T00:23:08.197 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-09-30T00:23:08.197 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.207 INFO:tasks.ceph.mds.b:Stopped 2023-09-30T00:23:08.208 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T00:23:08.208 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T00:23:08.208 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.260 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T00:23:08.260 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T00:23:08.260 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.303 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T00:23:08.303 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T00:23:08.303 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.334 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T00:23:08.334 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T00:23:08.335 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.401 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T00:23:08.401 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T00:23:08.401 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T00:23:08.401 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.425 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T00:23:08.425 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-09-30T00:23:08.425 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.467 INFO:tasks.ceph.mgr.y:Stopped 2023-09-30T00:23:08.468 DEBUG:tasks.ceph.mgr.z:waiting for process to exit 2023-09-30T00:23:08.468 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.535 INFO:tasks.ceph.mgr.z:Stopped 2023-09-30T00:23:08.535 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T00:23:08.535 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T00:23:08.535 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.545 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T00:23:08.545 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T00:23:08.545 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.595 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T00:23:08.595 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T00:23:08.595 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:23:08.606 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T00:23:08.606 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T00:23:08.609 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_' | egrep -v 'replacing it with standby' | egrep -v 'No standby daemons available' | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'objects misplaced' | egrep -v 'Synthetic exception in serve' | egrep -v 'influxdb python module not found' | egrep -v '\(MGR_ZABBIX_' | egrep -v 'foo bar' | egrep -v 'Failed to open Telegraf' | egrep -v 'evicting unresponsive client' | egrep -v '1 mgr modules have recently crashed \(RECENT_MGR_MODULE_CRASH\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-09-30T00:23:08.663 INFO:teuthology.orchestra.run.smithi094.stdout:1696033059.8740554 mgr.x (mgr.4106) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:23:08.663 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T00:23:08.663 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_' | egrep -v 'replacing it with standby' | egrep -v 'No standby daemons available' | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'objects misplaced' | egrep -v 'Synthetic exception in serve' | egrep -v 'influxdb python module not found' | egrep -v '\(MGR_ZABBIX_' | egrep -v 'foo bar' | egrep -v 'Failed to open Telegraf' | egrep -v 'evicting unresponsive client' | egrep -v '1 mgr modules have recently crashed \(RECENT_MGR_MODULE_CRASH\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-09-30T00:23:08.730 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_' | egrep -v 'replacing it with standby' | egrep -v 'No standby daemons available' | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'objects misplaced' | egrep -v 'Synthetic exception in serve' | egrep -v 'influxdb python module not found' | egrep -v '\(MGR_ZABBIX_' | egrep -v 'foo bar' | egrep -v 'Failed to open Telegraf' | egrep -v 'evicting unresponsive client' | egrep -v '1 mgr modules have recently crashed \(RECENT_MGR_MODULE_CRASH\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-09-30T00:23:08.798 INFO:teuthology.orchestra.run.smithi094.stdout:1696033059.8740554 mgr.x (mgr.4106) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:23:08.798 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi094.front.sepia.ceph.com 2023-09-30T00:23:08.798 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T00:23:09.016 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi121.front.sepia.ceph.com 2023-09-30T00:23:09.017 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T00:23:09.211 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi121.front.sepia.ceph.com 2023-09-30T00:23:09.211 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T00:23:09.301 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi121.front.sepia.ceph.com 2023-09-30T00:23:09.301 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T00:23:09.360 INFO:tasks.ceph:Archiving mon data... 2023-09-30T00:23:09.361 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690/data/mon.a.tgz 2023-09-30T00:23:09.361 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2023-09-30T00:23:09.377 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.Tmz6a20NQz 2023-09-30T00:23:09.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Tmz6a20NQz 2023-09-30T00:23:09.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.Tmz6a20NQz 2023-09-30T00:23:09.651 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.Tmz6a20NQz is 313KB 2023-09-30T00:23:09.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:23:09.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:23:09.786 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.Tmz6a20NQz 2023-09-30T00:23:09.800 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690/data/mon.c.tgz 2023-09-30T00:23:09.800 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2023-09-30T00:23:09.854 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.GH9uzY8eoM 2023-09-30T00:23:09.855 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.GH9uzY8eoM 2023-09-30T00:23:09.996 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.GH9uzY8eoM 2023-09-30T00:23:10.077 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.GH9uzY8eoM is 436KB 2023-09-30T00:23:10.173 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.GH9uzY8eoM 2023-09-30T00:23:10.187 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690/data/mon.b.tgz 2023-09-30T00:23:10.187 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2023-09-30T00:23:10.204 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.868iInGfw5 2023-09-30T00:23:10.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.868iInGfw5 2023-09-30T00:23:10.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.868iInGfw5 2023-09-30T00:23:10.450 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.868iInGfw5 is 436KB 2023-09-30T00:23:10.547 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.868iInGfw5 2023-09-30T00:23:10.560 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T00:23:10.561 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:23:10.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:23:10.624 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T00:23:10.624 DEBUG:teuthology.orchestra.run.smithi094:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:23:10.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:23:10.650 INFO:teuthology.orchestra.run.smithi094.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:23:10.688 INFO:teuthology.orchestra.run.smithi121.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:23:10.694 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T00:23:10.696 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690/remote/smithi094/crash 2023-09-30T00:23:10.696 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:23:10.723 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690/remote/smithi121/crash 2023-09-30T00:23:10.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:23:10.758 INFO:tasks.ceph:Shutting down logrotate 2023-09-30T00:23:10.758 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:23:10.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-09-30T00:23:10.845 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-09-30T00:23:10.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-09-30T00:23:10.893 INFO:tasks.ceph:Compressing logs... 2023-09-30T00:23:10.893 DEBUG:teuthology.orchestra.run.smithi094:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:23:10.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:23:13.388 INFO:tasks.ceph:Archiving logs... 2023-09-30T00:23:13.388 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690/remote/smithi094/log 2023-09-30T00:23:13.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:23:13.675 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690/remote/smithi121/log 2023-09-30T00:23:13.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:23:13.963 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:23:14.030 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a82f23d0a12c464d9955d79fc258d3c5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:23:14.032 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T00:23:14.041 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T00:23:14.048 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T00:23:14.048 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:23:14.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:23:14.144 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T00:23:14.152 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T00:23:14.153 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:23:14.155 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:23:14.168 INFO:teuthology.orchestra.run.smithi094.stderr:bash: ntpq: command not found 2023-09-30T00:23:14.169 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2023-09-30T00:23:14.172 INFO:teuthology.orchestra.run.smithi094.stdout:210 Number of sources = 4 2023-09-30T00:23:14.172 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:23:14.172 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2023-09-30T00:23:14.172 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 -477us[ -477us] +/- 78ms 2023-09-30T00:23:14.172 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -461us[ -447us] +/- 75ms 2023-09-30T00:23:14.173 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +900us[ +900us] +/- 67ms 2023-09-30T00:23:14.173 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:23:14.174 INFO:teuthology.orchestra.run.smithi121.stdout:210 Number of sources = 4 2023-09-30T00:23:14.174 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:23:14.174 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2023-09-30T00:23:14.174 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 -456us[ -456us] +/- 78ms 2023-09-30T00:23:14.174 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -467us[ -450us] +/- 75ms 2023-09-30T00:23:14.174 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +908us[ +924us] +/- 67ms 2023-09-30T00:23:14.175 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:23:14.175 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T00:23:14.182 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T00:23:14.183 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T00:23:14.192 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:23:14.309 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:23:14.400 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:23:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:23:14.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:23:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:23:14.460 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T00:23:14.469 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T00:23:14.477 INFO:teuthology.task.internal:Duration was 1106.219911 seconds 2023-09-30T00:23:14.477 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T00:23:14.485 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T00:23:14.485 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:23:14.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:23:14.529 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:23:14.544 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:23:14.768 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T00:23:14.768 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2023-09-30T00:23:14.768 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:23:14.787 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2023-09-30T00:23:14.787 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:23:14.805 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T00:23:14.805 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:23:14.829 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:23:14.883 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T00:23:14.883 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:23:14.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:23:14.993 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T00:23:15.002 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T00:23:15.002 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:23:15.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:23:15.050 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T00:23:15.059 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:23:15.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:23:15.089 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2023-09-30T00:23:15.113 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2023-09-30T00:23:15.127 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:23:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:23:15.157 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:23:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:23:15.181 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T00:23:15.190 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T00:23:15.191 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690/remote/smithi094 2023-09-30T00:23:15.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:23:15.233 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690/remote/smithi121 2023-09-30T00:23:15.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:23:15.267 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T00:23:15.267 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:23:15.269 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:23:15.315 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T00:23:15.325 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T00:23:15.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T00:23:15.365 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T00:23:15.365 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:23:15.367 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:23:15.379 INFO:teuthology.orchestra.run.smithi094.stdout: 262897 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:23 /home/ubuntu/cephtest 2023-09-30T00:23:15.381 INFO:teuthology.orchestra.run.smithi121.stdout: 262396 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:23 /home/ubuntu/cephtest 2023-09-30T00:23:15.382 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T00:23:15.392 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T00:23:15.411 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T00:23:15.436 DEBUG:teuthology.nuke:shortname: smithi094 2023-09-30T00:23:15.436 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:23:15.457 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 23:57:50.701634', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMqbQIXsPQL+Ar4JA2DrEuvSb1C56pvU9IeQTqdHEFydStZM3Jh68Ce792vRf9g1fPDGyeeDxExrc5ud827ofuI='} 2023-09-30T00:23:15.462 DEBUG:teuthology.nuke:shortname: smithi121 2023-09-30T00:23:15.463 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:23:15.483 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 23:57:50.703510', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG/8gFow/QHseLGfYRci2qn39BiLQNOI1wirw6FMtOPWoqfPf2NxV7uScS9oAVi0/HZb9BM2iGwESeViMP3MP7c='} 2023-09-30T00:23:15.502 INFO:teuthology.orchestra.console.smithi094:Power off 2023-09-30T00:23:15.503 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:23:15.525 DEBUG:teuthology.orchestra.console.smithi094:power off output: Chassis Power Control: Down/Off 2023-09-30T00:23:15.525 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:23:15.538 INFO:teuthology.orchestra.console.smithi121:Power off 2023-09-30T00:23:15.539 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:23:15.547 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2023-09-30T00:23:15.559 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2023-09-30T00:23:15.559 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:23:15.581 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-09-30T00:23:19.549 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:23:19.581 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:23:19.658 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2023-09-30T00:23:19.690 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-09-30T00:23:23.663 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:23:23.691 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:23:23.772 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is off 2023-09-30T00:23:23.798 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-09-30T00:23:23.874 INFO:teuthology.orchestra.console.smithi094:Power off completed 2023-09-30T00:23:27.802 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:23:27.912 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2023-09-30T00:23:28.013 INFO:teuthology.orchestra.console.smithi121:Power off completed 2023-09-30T00:23:28.117 INFO:teuthology.run:Summary data: description: rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{rhel_8} tasks/module_selftest} duration: 1106.219910621643 failure_reason: '"1696033059.8740554 mgr.x (mgr.4106) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.x: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a82f23d0a12c464d9955d79fc258d3c5 status: fail success: false 2023-09-30T00:23:28.117 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:23:28.197 INFO:teuthology.run:FAIL